BrandonFors/pandas-github-issues huggingface dataset:
This fixes issue 5797 and allows replace using timestamps. 
State: closed

this fixes https://github.com/pydata/pandas/issues/5797


Labels:

Comments:
can u add a release note (under bug fixes) and squash down the commits 

thxs

How would I edit the docs?

also, to squash down the commits do you simply use rebase -i?
 (Sorry, I am not a git power user.  we are a mercurial shop)

On Mon, Dec 30, 2013 at 7:08 PM, jreback notifications@github.com wrote:

> can u add a release note (under bug fixes) and squash down the commits
> 
> thxs
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5800#issuecomment-31377116
> .

see here: https://github.com/pydata/pandas/wiki/Using-Git

edit pandas/source/doc/release.rst

Jeff, 
Thanks for the help.  I had an issue for the extra commit from master.  I will issue a new pull request with one commit. 

in the future you can just rebase that away

git rebase -I origin/master

then just delete the offending commits

Thanks.  This was really helpful.    It will be much easier for me next
time.

On Mon, Dec 30, 2013 at 8:36 PM, jreback notifications@github.com wrote:

> in the future you can just rebase that away
> 
> git rebase -I origin/master
> 
> then just delete the offending commits
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5800#issuecomment-31379805
> .
TST: close sparc test failures
State: closed

TST: closes #5778, failing tests on non-little endian for stata (spac)
TST: closes #5779, big endian compensation for datelike dtypes


Labels:

Comments:
Those tests indicate genuine broken code right? will it throw if a user tries to use the 
stata functionality on sparc?

@y-p not really sure. it may be a function of the test, because most of the tests actually work, I think its a locale/unicode issue on stata for big endian. you are welcome to take a look.

@y-p i created #5781 to track the stata failures (that are skipped now on big endian) if anyone has time

great, that works.
BUG: IndexError on read_csv/read_table when using usecols/names parameters and omitting last column
State: closed

This addresses an issue in read_csv/read_table where there is no header and both usecols and names are assigned but the last column is not included. This caused an IndexError after reaching the last column specified in usecols. Existing test cases for `usecols` were modified to expose and address this specific issue.
#5766


Labels:

Comments:
superceded by #5770,  I cherry picked the test, Thanks
DOC: Flesh out the R comparison section of docs (GH3980)
State: closed

Some additions to #3980. 

I've done some more comparisons to the plyr functions located [here](http://nbviewer.ipython.org/gist/chappers/8066230/). More specifically [aaply](http://nbviewer.ipython.org/gist/chappers/8066230/aaply.ipynb), [alply](http://nbviewer.ipython.org/gist/chappers/8066230/alply.ipynb) don't really have much to do with pandas (its just list comprehension). and [dlply](http://nbviewer.ipython.org/gist/chappers/8066230/dlplyr.ipynb) is probably more suited to be in statsmodel example rather than pandas. 


Labels:

Comments:
Thanks @chappers , this is useful stuff. But the PR violates a bunch of git-related guidelines
explained in [CONTRIBUTING.md](https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md):
- You merged master into your PR branch.
- You developed the PR on top of a pandas commit from 6 months ago, that's about 2500 commits 
  back in the fog of time.

Both of these do nasy things to the git history and so the devs follow a bunch of rules on how to use git to avoid that.

Please fix it yourself or let us know if you need git help. It's fine if you do, we all did at some point.
We value your work, we want your help making pandas better, but please follow the guidelines or ask 
for help if you need it.

@chappers - easiest is just to copy paste your changes to a separate file, checkout current master, and then paste them back on. Since it's docs _and_ a relatively new section of the docs, you should be fine.

Yeah...I don't have any idea what I'm doing. So I'm suppose to update a branch to upstream/master and then run `git rebase` on it? And then submit a PR?

No, that would have prevented this, now you need to fix the existing branch.

upddate your master branch to point at current master on github, then
checkout your PR branch. then:

```
[chappers-r-compare] % git reset master    # reset head, your "starting point", without changing the working copy
Unstaged changes after reset:
M   doc/source/comparison_with_r.rst
[chappers-r-compare] % ga doc/source/comparison_with_r.rst
[chappers-r-compare] % gc -m "DOC: your description here"
# and finally force push to clobber the current PR branch on github with the repaired one
git push -f
```

Thanks a lot.

nicely done.

Thanks for this work! 
I add some comments inline.

since this is only docs, as soon as @jorisvandenbossche , @y-p ok...then can merge it in

I haven't run sphinx, but assuming it's ok... merge away.

I ran Sphinx with this, and its OK. I cannot judge everything in the PR (the R code), but it seems OK.

ok will merge and can check how 

@chappers

docs look good

http://pandas.pydata.org/pandas-docs/dev/comparison_with_r.html

feel free to add more :)
Update rolling skew & kurtosis to handle cases where they aren't defined
State: closed

closes #5749

The rolling skewness and kurtosis in algos.pyx were modified to match
the testing logic in pandas/core/nanops.py. They now both return NaN
where they are not defined, which occurs where there are either too
few observations or where the variance is zero.

A set of tests was added to verify that Nan is returned in these cases
and that the computations continue to work correctly when the values
are defined.


Labels:

Comments:
pls add a mention in Bug Fixes in 0.13.1...thxs

otherwise looks ready to go

I made those changes. I noticed there were already tests for the rolling moments but they were geared towards randomized series of acceptable length, so I added these tests separately - I gave them the umbrella title "edge cases". Is that sufficient?

@yieldsfalsehood perfect thanks

just need to rebase away the track branch...so this:

```
git fetch upstream
git rebase -i upstream/master
```

delete the merge remote-tracking branch

then

```
git push upstream thisbranchname -f
```

see here: https://github.com/pydata/pandas/wiki/Using-Git

Woops! Sorry.

pls rebase and then I think can mergge this

There we go. Sorry - first PR. Thanks for the patience.

thanks!
ENH: set display.max_seq_items default != None
State: closed

closes #3391.

No idea what the default value should be. numpy equivalent default is 1000, but even 100 is too big IMO. 
The default is primarily there to mitigate output bombs when displaying some pandas objects.

Still need to update the release notes on all these micro-PRs.


Labels:

Comments:
@y-p I think this ought to go in 0.13

I'm down with that. I'll fix the broken tests and add some docs.

Added docs and hopefully green. Last chance to bikeshed on default value of 100.
ENH: expose option_context as a top-level API GH5618
State: closed

<del>
Moving `option_context` to toplevel rather then making `set_option` a context manager means
a one line change instead of metaclasses and config_prefix subtlety and the rest of the ich.
When a given approach to something explodes in complexity I'll [bravely, bravely run away](http://www.youtube.com/watch?v=BZwuTo7zKM8) every single time.
</del>


closes #5618
replaces #5625. 

@jtratner, your points about config_prefix being broken are valid, feel free to pick up in the future
if you're so inclined.

cc @jseabold


Labels:

Comments:
Why introduce a new name into the name space when you can just add `__enter__` and `__exit__` methods to `set_option`? I hit the meta class because I was doing too many things at once... 

I didn't understand all the crud you had to work around until I tried it myself.
it makes the config_prefix stuff break and it makes the code feel like doing a summersault 
while reciting shakespeare backwards while doing taxes. It just makes me want to stand back.

The devs have been fine with this distinction for a long while, why's it not good enough for users?

My bad, I misinterpreted the failing tests. Hopefully all green now with `set_option` as context manager.
Corrected authorship as well.

Okay, sure. Just seems like you could do
`s/set_option/_prefixed_set_option/g` and then rename `option_context` to
`set_option` (and if you use it not as a contextmanager it's exactly the
same as `set_option`)
On Dec 19, 2013 6:05 PM, "y-p" notifications@github.com wrote:

> I didn't understand all the crud you had to work around Until I tried it
> myself.
> it makes the config_prefix stuff breaks and the summersault while reciting
> shakespeare
> backwards and doing taxes feel of the other out just makes me want to
> stand back.
> 
> The devs have been fine with this distinction for a long while, why's it
> not good enough for users?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5752#issuecomment-30975432
> .

Hmm, thought I did, I basically copy-pasted your code.
The docstrings for != set_option didn't work btw, the @property thing doesn't work
as expected for functions. I had to put in a fix. 

The option_context thing is just cleaner all around, but I think it's all there now.

Go with whatever you like best. option_context is a much better name for it.

Opted for exposing `option_context` as top-level API. It's just a simpler solution.
BUG: don't allow an empty dataframe to have scalar assignment succeed (GH5744)
State: closed

closes #5744
related #5720

I think this is the correct behavior

```
In [1]: df = pd.DataFrame(columns=['a', 'b', 'c c'])

In [2]: df['d'] = 3
ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

In [3]: df['c c']
Out[3]: Series([], name: c c, dtype: object)
```


Labels:

Comments:
Can we just allow the scalar assignment for an empty dataframe? 
It can be useful to add new columns dynamically.
A common use case for me is to read a data file using `read_csv` and then add some derived columns.
Checking that the read dataframe is not empty everytime before adding a new column could be tedious.

@socheon 

in general working with a dataframe that does not have an index is very odd to begin with
if you have an index, then their is no problem

you can `read_csv` and then add a column with a scalar no problem

if you have an example of something you think ought to work, pls put it up

@jreback 

At work, I receive daily many third party csv files (e.g. new orders etc) which I need to parse, add some derived columns, and then load into database. Occasionally, some of the files may be empty and just having the column names. For example

```
Date,ProductID,Quantity,UnitCost
01282013,1,10,3
01282013,2,5,6
```

To load into database, I simply use

```
from datetime import datetime
df = pd.read_csv('data.csv', dtype={'Date': 'object'})
df.Date = df.Date.map(lambda s: datetime.strptime(s, '%m%d%Y'))
df['TotalCost'] = df.Quantity * df.UnitCost
load_into_database(df)
```

But on some days when there are no orders i.e. the input file is just

```
Date,ProductID,Quantity,UnitCost
```

The program would break when adding new columns to empty dataframe. Of course, I could always
check `if len(df) > 0` before adding new columns. But somehow, I would still need to add the `TotalCost` column to the empty dataframe before passing it to the function `load_into_database`.

Anyway, the intended change will not a problem for me. Probably this example is too specific and has easy workaround.

@socheon 

I hear ya!

just trying to guard against have an operation which looks like it succeeds, but actually doesn't do anything.

but I see that in 0.12 that was ok....

let me see...maybe I will revert back..

@socheon alright...give a try with master again...and let me know (original behavior is restored)

Thanks. I tried again and the master is good.
Fix prefix argument for read_csv/read_table
State: closed

Closes #5732 


Labels:

Comments:
can you add some tests for this?

One question I had about this - while use of the `prefix` argument now matches the documentation, this change is no longer backward compatible (if a user passed an arbitrary string and expected the columns to be named X1, X2,...)

@nmichaud prob just put a note in v0.14.txt to note that

OK, added a test for the prefix argument.

@nmichaud actually just add a release notes entry in API for 0.13...and i'll throw it in...thanks

@jreback, I'll add the release notes, you vetted this?

Will push release note directly.
DOC: DatetimeIndex accepts name param
State: closed

Just a docstring change to reflect `DatetimeIndex` taking a name parameter. I added a test since it wasn't explicitly tested anywhere.

``` python
In [1]: from pandas import DatetimeIndex

In [2]: idx = DatetimeIndex(start='2000-01-01', periods=1, freq='A', name='TEST')

In [3]: idx.name
Out[3]: 'TEST'
```


Labels:

Comments:
cheers.
BUG: Bug in fillna with Series and a passed series/dict (GH5703)
State: closed

closes #5703

```
In [1]: s1 = Series([np.nan])

In [2]: s2 = Series([1])

In [3]: s1.fillna(s2)
Out[3]: 
0    1
dtype: float64

In [4]: s1.fillna({ 1 : 1})
Out[4]: 
0    1
dtype: float64

In [5]: s1.fillna({ 1 : 2, 2 : 3})

ValueError: cannot fillna on a 1-dim object with more than one value
```


Labels:

Comments:
@jreback I think with this change also the docstring should be updated? Because a Series is not mentioned now.

hmm thought it was

@jorisvandenbossche docs updated in #5716 

@jreback Super!
Add cdfplot
State: closed

WIP for now. Closes #2669

I'm using statsmodels' KDE implementation right now; it has a `cdf` method, but scipy's gaussian_kde doesn't. I need to check the math, but I think something like doing a cumsum on the density and normalizing by the sum should be the same.

I'm also going to add kwargs for things like the inverse cdf. 

![cdf](https://f.cloud.github.com/assets/1312546/1749251/0287778c-6505-11e3-9dae-cc9e9c37a1de.png)


Labels:

Comments:
@TomAugspurger not saying the dict thing is required to get this merged, but if you wanted to add a second commit that changed this all to two dicts (one for Series, one for DataFrame), rather than if/elif statements, that'd be a complementary addition to this PR.

FWIW, in statsmodels (and places in scipy.stats), we compute the cdf from the pdf using `scipy.integrate.quad`.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/nonparametric/kde.py#L161

Okay this one can sit until 0.14 has started and I can add a release notes entry. I got on a bit of DRYness run, so I ended up combining the `KdePlot` and `CDFPlot` into a `DistributionPlot` class.

I've only used statsmodels for the actual fitting of the cdf. Scipy doesn't provide a cdf on their kde object, so we'd have to duplicate the code that @jseabold linked to above. May as well just use what they've done.

And of course if this has fallen out of scope for pandas then close without merging!

We can't have a dependency on statsmodels, except for testing.
I remember `kde` plots were a bit suspect in terms of scope too,
expanding further is therefore suspect too.

Let's call this out of scope.
DISP: show column dtype in DataFrame.info() output
State: closed

closes #3429.

Old:

```
In [39]: DataFrame(randn(10,2)).info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10 entries, 0 to 9
Data columns (total 2 columns):
0    10  non-null values
1    10  non-null values
dtypes: float64(2)
```

New:

```
In [39]: DataFrame(randn(10,2)).info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10 entries, 0 to 9
Data columns (total 2 columns):
0    10 non-null float64
1    10 non-null object
dtypes: float64(1), object(1)
```


Labels:

Comments:
:+1: 

@cpcloud can you put an example of old and new up in the descr...

sure thing

@y-p @jreback merge on passing?

I think their is an issue about adding the info method to Series, so maybe make a quick test method that assets that it raises (and ref this issue) - because series doesn't have a dtypes method (and this needs a modification for this change)

ok

0.14 starts next week, honor the bug-fix only period?

@y-p ahh yes... sorry. merge after 0.13 rls

It's clear nothing could possibly go wrong with something as simple as this. We've been there before... :)

@jreback 

> and this needs a modification for this change

not sure what u mean

what I mean is that the implementation would break for an info method on series (which doesn't exists at the moment)

so I think that you should have a test of this method that assert raises with a series for attribute error

Yep ok that's what I thought. Test is there.

On Sunday, December 15, 2013, jreback wrote:

> what I mean is that the implementation would break for an info method on
> series (which doesn't exists at the moment)
> 
> so I think that you should have a test of this method that assert raises
> with a series for attribute error
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5682#issuecomment-30606978
> .

## 

## 

Best,
Phillip Cloud

@jreback , can I merge this or is it pending #5167?

i think this is fine

Thanks @cpcloud. You used to be a code fountain not that long ago, did RL catch up with you? :)

will add release notes.

Got a job where I don't use Python. Had to put my pandas love on the back burner while I refamiliarize myself with C++ and learn a bunch of windows OS material. Not a whole lot of time on Linux. It brings me to tears occasionally. I'll try to at least bring it to a trickle in the coming months. 

please. trickle. :)

if you're in microsoft land, LINQ and Rx are quite beautiful...

I can add those release notes tomorrow if you don't beat me to it. Otherwise, much appreciated. 

Done 10 minutes ago, don't worry about it.
API/ENH: Detect trying to set inplace on copies in a nicer way, related (GH5597)
State: closed

related #5597
pretty common error that I have seen, originally from:

http://stackoverflow.com/questions/20508968/series-fillna-in-a-multiindex-dataframe-does-not-fill-is-this-a-bug

```
In [1]: a = [12, 23]

In [2]: b = [123, None]

In [3]: c = [1234, 2345]

In [4]: d = [12345, 23456]

In [5]: tuples = [('eyes', 'left'), ('eyes', 'right'), ('ears', 'left'), ('ears', 'right')]

In [6]: events = {('eyes', 'left'): a, ('eyes', 'right'): b, ('ears', 'left'): c, ('ears', 'right'): d}

In [7]: multiind = pandas.MultiIndex.from_tuples(tuples, names=['part', 'side'])

In [8]: zed = pandas.DataFrame(events, index=['a', 'b'], columns=multiind)
```

Shows a warning (the message is comewhat generic)

```
In [9]: zed['eyes']['right'].fillna(value=555, inplace=True)
/usr/local/bin/ipython:1: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
  #!/usr/local/bin/python

In [10]: zed
Out[10]: 
part  eyes         ears       
side  left  right  left  right
a       12    123  1234  12345
b       23    NaN  2345  23456

[2 rows x 4 columns]
```

Correct method

```
In [11]: zed.loc[:,('eyes','right')].fillna(value=555, inplace=True)

In [12]: zed
Out[12]: 
part  eyes         ears       
side  left  right  left  right
a       12    123  1234  12345
b       23    555  2345  23456

[2 rows x 4 columns]
```


Labels:

Comments:
cc @michaelaye .... I know you use lots of multi-index stuff...can you give this a quick test run and lmk if any SettingWithCopyError/Warns come up (that are unexpected). ? thxs

I tested 222bc9d and got no Error/Warns.
But I got to warn you that I don't have much multi-index stuff anymore. The issues we discussed recently were months old, and because of the pending issues that I partly didn't understand, and because they are harder to grasp in general I refrained from using them and solved my stuff mostly with 1-D indexes.

@michaelaye gr8 thanks. It was also trying to do inplace operations when operating on a copy. So you code seems good.
BUG: Work-around a numpy regression affecting pandas.eval() with numexpr
State: closed

numpy 1.7.0 erroneously raises IndexError instead of ValueError
from ndarray.item() when the array is not of length 1. This can be
seen as a failure of

```
computation.tests.test_eval.TestScope.test_global_scope
```

for the cases that engine='numexpr'.

Absorb the splatter from this regression by explicitly catching the
erroneous IndexError.

closes #5535.


Labels:

Comments:
weird that this is only on FreeBSD. any idea why?

I don't think it is actually limited to FreeBSD. The affected code in numpy does not appear to be platform-dependent. I think that we just didn't see test failures, because as far as I can tell (pydata/pandas@5f4be91f, pydata/pandas@81739d9c) we've never actually encountered the affected version of numpy in the Travis-CI environment. The fix was backported to the 1.7.x maintenance branch prior to 1.7.1 (numpy/numpy@b84555ac).

ahh that makes sense

but this is a fix for a version that is already fixed then in numpy,  why is it necessary to fix something that doesn't occur?

It does occur if you're in an environment where 1.7.0 is present, like FreeBSD. Pandas may decide not to accept this patch as a matter of policy (e.g. we don't support numpy 1.7.0), in which case I could elect to apply the patch to the FreeBSD port. My aim, though, is to minimize the diff between the ported software and the published releases, so I would like to see this patch accepted.

@neirbowj 

maybe I was not clear. Is the current numpy 1.7.0 patched? you said it was back ported? Or if you happend to have an 'older' numpy 1.7.0 then it (obviously) does not have the back port.

@jreback 

Ah, I see now. No, numpy 1.7.0 does not have the fix and is broken. 1.7.1 is fixed and 1.8.0 is fixed.

ahh..ok...then i don't see a problem with this (as it seems to work with 1.7.1/1.8 which we test)...

thanks for the fix

can you add a release note referecing the issue?

thanks for the fix!

@neirbowj, are you in a position to put up a freebsd CI server / buildbot for us to track? 

@y-p, That's a compelling possibility. I will look into it.
BUG/API: autocorrelation_plot should accept kwargs
State: closed

Unless there's a specific reason not to for this plot?  All the others seem to accept them.

``` python
autocorrelation_plot(data, label='ACF')
```

![label_autocorr](https://f.cloud.github.com/assets/1312546/1649385/4cad26d2-59fb-11e3-9d3e-2ccc7c44856e.png)


Labels:

Comments:
can you rebase, and move release notes to 0.13.1..thxs

@TomAugspurger can you rebase and move notes to 0.13.1..thxs

@jreback rebased and moved to 0.13.1. Sorry I missed the earlier ping, I think my email settings were messed up.

This whole plotting function needs a bunch of work, but I guess this is a patch. I'll try to get back to this later.

thanks @TomAugspurger pls look thru plotting issues at your leisure and do some PR's!