BrandonFors/pandas-github-issues huggingface dataset:
BUG FIX: wide_to_long modifies stubnames
State: closed

Simple fix that prevents modification of the list `stubnames`, which is an argument to `wide_to_long`. 

Closes #9204


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
This needs a test and an addition to the bug fixes section of what's new for 0.16.

I have never written a test. I see you have a test written already for wide_to_long:

``` python
class TestWideToLong(tm.TestCase):
    def test_simple(self):
        np.random.seed(123)
        x = np.random.randn(3)
        df = pd.DataFrame({"A1970" : {0 : "a", 1 : "b", 2 : "c"},
                           "A1980" : {0 : "d", 1 : "e", 2 : "f"},
                           "B1970" : {0 : 2.5, 1 : 1.2, 2 : .7},
                           "B1980" : {0 : 3.2, 1 : 1.3, 2 : .1},
                           "X"     : dict(zip(range(3), x))
                          })
        df["id"] = df.index
        exp_data = {"X" : x.tolist() + x.tolist(),
                    "A" : ['a', 'b', 'c', 'd', 'e', 'f'],
                    "B" : [2.5, 1.2, 0.7, 3.2, 1.3, 0.1],
                    "year" : [1970, 1970, 1970, 1980, 1980, 1980],
                    "id" : [0, 1, 2, 0, 1, 2]}
        exp_frame = DataFrame(exp_data)
        exp_frame = exp_frame.set_index(['id', 'year'])[["X", "A", "B"]]
        long_frame = wide_to_long(df, ["A", "B"], i="id", j="year")
        tm.assert_frame_equal(long_frame, exp_frame)
```

Could I just build an assertion within this existing test where I check the stubnames before and after the function call?

And how do I add to the bug fixees section? I do not think I saw in the developer's guide for contributing to pandas.

@jtorcasso ideally just add another tests just like this one, but that asserts the stub names don't change (you could modify this one as well, whatever works). Then add in doc/source/whatsnew/0.16.0.txt a note

@jtorcasso to be concrete, you can add another `test_...` function (like `test_simple`) to the `TestWideToLong` class. 
You can eg just use the example you showed in the issue (#9204) that failed, asserting that the stubs list in the end is still the same.

Hopefully this last commit did the job.

merged via https://github.com/pydata/pandas/commit/fda50121453f76142b00ff57b017b8a3ef692f69

thanks!
added mising numeric_only option for DataFrame.std/var/sem
State: closed

closes https://github.com/pydata/pandas/issues/9201, the `numeric_only` option is missing for `DataFrame.std()` (and also `DataFrame.var()` and `DataFrame.sem()`), this is a fix for it


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
pls add a release note in the Bug Fix section of 0.16

@jreback updated, please take a look, thanks 

@jreback PR updated, please take a look

I think that if u have a string 100 in a column and the rest are actual float/int values (thus dtype is object) thrn this should always raise a ValueError and not do the coercion

it hides errors to evaluate the string as a number

that's a good point. I'll dig into this. 

@mortada pls rebase and update according to the last. thanks

pls rebase

@jreback just rebased, sorry I've been pretty swamped and haven't got around to this. But I should have more time going forward. 

@jreback rebased and addressed the issue you mentioned about type coercion. Please take a look. 

With this PR `_nanvar` will only convert values for `int` types, and not coerce `str` or other types into `float` like it used to. It will throw a `TypeError` if `numeric_only` is `False` and there's a `str` type.

This is now consistent with the behavior of other methods such as `DataFrame.mean`

thanks!
9203 SQUASHED - DOCS: doc string edited pandas/core/frame.duplicated()
State: closed

Redefined `take_last` variable in doc string. Original definition only
made sense for drop_duplicates()


Labels:
name: Docs

Comments:
@robertdavidwest Thanks!

For future reference, it is not needed to create an new branch/new PR when squashing. If you just do 

```
git rebase -i upstream/master
```

and then squash, you can just push the same branch again (but using `--force`):

```
git push --force origin DataFrame_duplicated_docstring_amend
```

Thanks Joris. baby steps...

On 7 January 2015 at 11:28, Joris Van den Bossche notifications@github.com
wrote:

> @robertdavidwest https://github.com/robertdavidwest Thanks!
> 
> For future reference, it is not needed to create an new branch/new PR when
> squashing. If you just do
> 
> git rebase -i upstream/master
> 
> and then squash, you can just push the same branch again (but using
> --force):
> 
> git push --force origin DataFrame_duplicated_docstring_amend
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9206#issuecomment-69047527.

No problem! :-)
BUG: fix for GH9010
State: closed

Superseded by #9358 

---

This is an update of PR #9024.

closes #9010 


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@PollyP can you rebase, and update according to comments by @shoyer 

OK, will do by Saturday at the latest.

On Sun, Jan 18, 2015 at 1:00 PM, jreback notifications@github.com wrote:

> @PollyP https://github.com/PollyP can you rebase, and update according
> to comments by @shoyer https://github.com/shoyer
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9198#issuecomment-70425684.

Looks like dstephens99 went ahead and fixed this. 

yes, @dstephens99 fixed it in pandas-datareader, but until we decide what exactly to do with the code in pandas, this can maybe still be merged in pandas? (@jreback)

(only reopened to be sure we decide on this, we can just close it again if we are not going to merge this)

see comments here: https://github.com/pydata/pandas-datareader/issues/15

@jorisvandenbossche are we merging this?

I think we should close this in favor of #9358 (which brings the changes made in pandas-datareader to back to pandas, including this)

@PollyP Thanks for your work on this! If you want to do more fixes in the future, you can submit them directly to pandas-datareader.
fixes division by zero error for kurt()
State: closed

Currently if `kurt()` is called on a `Series` with equal values, it throws a `ZeroDivisionError`

```
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: s = pd.Series(np.ones(5))
In [4]: s.kurt()
ZeroDivisionError: float division by zero
```

This is not consistent with the case when `kurt()` is called on a `DataFrame` of equal values

```
In [5]: df = pd.DataFrame(np.ones((5, 5)))
In [6]: df.kurt()
Out[6]:
0    0
1    0
2    0
3    0
4    0
dtype: float64
```

with this patch `s.kurt()` will return `0` instead of throwing.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
This looks good to me.

@mortada could you please add a note to the what's new for 0.16? You can reference the issue number for this PR.

@shoyer sure I've added a note to whatsnew for 0.16, this PR is updated.

is skew ok? are there similar tests? if not can you add 

@jreback `skew()` doesn't have this problem.

```
In [4]: for i in range(1, 5):
    print 'i is %d, skew is %f' % (i, pd.Series(np.ones(i)).skew())
   ...:
i is 1, skew is nan
i is 2, skew is nan
i is 3, skew is 0.000000
i is 4, skew is 0.000000
```

and sure I will add similar tests to `skew`

@jreback PR is updated, please take a look 

Lgtm

Thanks!
Documentations: Ecosystem/API - pydatastream (issue #5630)
State: closed

Following an old issue #5630 (Documentation / Remote Data Access - 3rd pary libraries).

Quandl was already described in Ecosystem / API section, so I've added link to pydatastream that targets Thomson Reuters Datastream.

closes #5630.


Labels:
name: Docs

Comments:
Does this close  #5630 fully?

@jorisvandenbossche - OK, fixed

I think it fully closes #5630.

As a second thought, I'm not sure if it makes sense to add support to Quandl/python or pydatastream directly in web.DataReader. It will amount to 

```
import quandl
return Quandl.get(...)
```

or 

```
from pydatastream import Datastream
DWE = Datastream(username="...", password="...")
return DWE.fetch(...)
```

What do you think?

I also think including this in pandas is not needed. First you raise the point of the non-free data source. And if the packages have a nice interface to get these data as you show above, they can certainly be a package on their own. We are also thinking of rather moving some of the data readers included in pandas to separate packages, rather than including more of them (see #8961). It is very easy to install eg the quandl package.

Then, for this PR: we always ask to squash the commits into one commit (you can do this with `git rebase -i upstream/master`) for small changes. Can you do this? 
If problems, just ask, or I can also do it on merge.

Just to be sure: after rebase I'm getting another branch (as the first one was already pushed to github).
Do I need to submit another PR from the new branch?

Normally, the command I gave does rebase your existing branch (doc_ecosystem_api), so you can just push it again to your fork, and this PR will automatically be updated (you will only have to 'force push': `git push --force origin doc_ecosystem_api`

Aha, force push! Thanks, I did not know about it

Done then

OK, thanks!

And if you are interested more in general in the data readers, certainly chime in on the linked issue #8961
BUG: read_html with a single column table #9178
State: closed

This commit fixes a crash on read_html(html, flavor="bs4") if table only has one column.

closes #9178 


Labels:
name: Bug
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
pls add a test

Please add a test

The test should probably test that if it succesfully parses one column table but in which file should I put it? I've never written nose tests.

Usually tests go in the tests sub-directory and in the module you want to test.  For example, you should put your test in `pandas.io.tests.test_html`.  And in there probably add a test in the `TestReadHtml` class.  There are plenty of examples there to get you started.  And then you can run your test using nosetests (you probably just want to run the module: `nosetests pandas.io.tests.test_html`).

@boarpig needs a test.

@boarpig this needs a test pls.

@boarpig pls add a test

closing, if/when updated pls reopen

I have just encountered this bug in 0.18 and the fix from this PR fixed that. Is the only limiting factor to merge this PR adding tests? I could probably do it (and will appreciate any help).

yep needs tests

But I can see there are already tests for this part [here](https://github.com/pydata/pandas/blob/827745dcf0cec55e8b071acc54b0aa5e610e640e/pandas/io/tests/test_html.py). What else should be added? This change seems to be sufficiently tested by current tests. Thanks

I am willing to finish this, but I need an answer to the previous post. 

you need to replicate the example in the issue - that is the test

I have done it in a [new PR](https://github.com/pydata/pandas/pull/12975) (so I don't have to do a rebase).
Add lag parameter to autocorrelation
State: closed

Add lag parameter to autocorrelation, default to lag-1 autocorrelation so existing code will work unchanged.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
This is not good! Failure unrelated to my commit:

# 

FAIL: test_tquery (pandas.io.tests.test_sql.TestXMySQL)

---

Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/io/tests/test_sql.py", line 2238, in test_tquery
    tm.assert_series_equal(result, expected)
  File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 674, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas/src/testing.c:2556)
  File "testing.pyx", line 93, in pandas._testing.assert_almost_equal (pandas/src/testing.c:1794)
  File "testing.pyx", line 140, in pandas._testing.assert_almost_equal (pandas/src/testing.c:2390)
AssertionError: expected -1.00016 but got -1.00015, with decimal 5

Have you seen this before?

@omtinez pls add a release note 0.16.0 enhancements section. and then squash. looks ok otherwise.

ok this seems ok. pls rebase on master and squash. ping when green.

Sorry about the mess, new to Git. I squashed the changes and now both autocorrelation-lag and master in my repo should reflect all of my desired changes.

@omtinez looks like you may have closed the pull request on accident. Could you reopen it?

For some reason, github closed my pull request and now sees zero commits in my branch. I tried doing a git push -f just to see if github would pick up my commit then but no dice: https://github.com/omtinez/pandas/commit/25b6b3747dd9835ad18350ea4717e91a16619a72

Github won't let me reopen unless it sees the commit, is there a way to fix this or do I need to open a new pull request?
Docstring: pd.to_datetime (issue #9107)
State: closed

Following the issue "to_datetime returns NaT for old dates with coerce=True".

Do I understand correctly that online documentation is build automatically from docstrings? I.e. are there any other files to change?

Also it would be good to make a link to [this section of "Caveats and Gotchas"](http://pandas.pydata.org/pandas-docs/version/0.15.0/gotchas.html#timestamp-limitations), but I'm not sure if the docstring is a good place to do this.


Labels:
name: Docs

Comments:
The online documentation is indeed built automatically from docstrings, but it doesn't exclude code in the examples:
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html#pandas.to_datetime

So it would nice to add the output to these examples (as well as the missing output for the previous examples in this docstring).

@shoyer - OK, corrected

Thanks!
ENH: Handle categorical dtype to/from R
State: closed


Labels:
name: Categorical
description: Categorical Data Type

Comments:
Failure is unrelated and why didn't skip-ci work?

Is using the pandas interface the preferred way to do pandas/R dataframe conversion, or should we leave this to rpy2 now that it has built in support for pandas? I've had pretty good luck with using `pandas2ri.activate()` from rpy2, which I believe entirely skips the conversion machinery in pandas itself. At the very least, I suspect a documentation update is due.

@lgautier any input?

For reference, here is where you'll find the pandas conversion code in rpy2: https://bitbucket.org/rpy2/rpy2/src/f1f15fe7eb6d1d65e8b3de4ee7fd697b800f001e/rpy/robjects/pandas2ri.py?at=default

BTW, I restarted the failed travis job -- we'll see if it work this time.

I am adding the following comment so you (`pandas`) can make the your decision with data.

`rpy2` has integrated a two-way conversion of `pandas` `DataFrame` to `rpy2` `DataFrame` (proxy of `data.frame` in the embedded R) since version 2.4.0 (current release is 2.5.4) and there is currently no plan to drop support for it: `rmagic`, originally in `ipython` and now part of `rpy2`, appears to rely on `pandas` for a prime representation of data frames in Python (@davclark can comment or add anything I'd have omitted about this).

While the conversion in `rpy2` is covered by unit tests and is thought to be practical and well behaved, `rpy2` is modular and relatively easy to customize by design. Alternative conversions, built from scratch or built on the top of the existing ones, can be added (see http://rpy.sourceforge.net/rpy2/doc-2.5/html/robjects_convert.html?highlight=register#customizing-the-conversion)

I don't have a strong opinion on this, though I suspect any "fixes" should definitely make it in to rpy2. As a prior though, I assume pandas development moves a bit faster, but that's mostly a guess. My mercurial/bitbucket-fu is also super weak (so far), so it's easier for me to make a pandas PR :)

Related discussions in #7309 and #7385

@jseabold it does not appear to me that any of this is actually tested on travis (it is all skipped), though it doesn't even print the skip message.

So ok with putting this in if you can confirm that it works locally for you?

cc @unutbu can you give a check?

@jseabold 

>  As a prior though, I assume pandas development moves a bit faster, but that's mostly a guess.

Ya ya pandas is more and better in any possible and conceivable way [yawn]
;-)

> My mercurial/bitbucket-fu is also super weak (so far), so it's easier for me to make a pandas PR :)

1- Go to https://bitbucket.org/rpy2/rpy2
2- Click "Fork" (left side, 5th icon)
3- Edit in the web browser
4- Submit a pull request

A use case for this PR: http://stackoverflow.com/q/28353937/190597

let's redirect to `rpy2` for these types of conversions, see #9602
is_list_like() for to_excel, fixes #8188. 
State: closed

closes #8188 


Labels:
name: Enhancement
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO Excel
description: read_excel, to_excel

Comments:
your comparison is going to be different as this is not a round-trippable format (e.g you will need to parse the return string and store it as a list in the expected frame).
- need an addition to the doc-string on the proposed conversion
- need a release note 

Okay, I will try and add those soon. Sorry about the incorrect test, I'm still learning on this stuff. 

@hunterowens you might want to check out the performance impact of this as well, given that it has to be called on every single value of every cell that gets written.

@hunterowens working on this?

closing pls reopen if/when updated
BUG: "index_col=False" not working when "usecols" is specified in read_csv (GH9082)
State: closed

Fixes #9082 
My access to the internet is spotty for the next few days. This fixes 9082 and does not break any of the existing tests so hopefully it won't break anyone's existing code.  Happy New Year!


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
does this also fix #9098 ? 

No.  I can reproduce the problem in 9098 even with my patched version. I will try to fix it this weekend

merged via https://github.com/pydata/pandas/commit/22f148620b18bb373412e42d399a97a5958a6312

thanks!
BUG: Fix for Timestamp handling in xlwt and xlsxwriter engines.
State: closed

Fix for writing Timestamp objects using the xlwt and xlsxwriter
engines. Both modules write Excel dates and times using
datetime.timedelta which differs from pandas.Timedelta. This
fix coerces Timestamp objects to datetime objects.

fixes #9139.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
ok, this seems ok. pls add a release note (bug fix section).

@jreback,

I've added a release note. Note 100% sure if it is in the correct place though. That is probably something that could be added to the Contributing.md doc as some stage.

If it is wrong let me know and I'll fix it.

Rebased and updated.

John

Release note is fine!

see #9185. Going to steal your tests, but should work after that fix is done/merged.

replaced by #9257 (tests only are necesssary)
ERR: better concat error messages #9157
State: closed

closes #9157 

This is a first attempt to improve the error message when trying to concat an empty list/set of objects:

`pd.concat([])` currently raises following error:

```
/Users/ch/miniconda/envs/sci33/lib/python3.3/site-packages/pandas/tools/merge.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy)
    765 
    766         if len(objs) == 0:
--> 767             raise ValueError('All objects passed were None')
    768 
    769         # consolidate data & figure out what our result ndim is going to be

ValueError: All objects passed were None
```

After the tiny fix it is:

```
/Users/ch/repo/pandas/pandas/tools/merge.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy)
    767 
    768         if len(objs) == 0:
--> 769             raise ValueError('No objects passed')
    770 
    771         if isinstance(objs, dict):

ValueError: No objects passed
```


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
needs a test

Yes, still thinking how to design a good test. I still raise a ValueError, but this time with a slightly different error message. Should I test for the wording of the error message? I am not too happy with this solution since it breaks if someone decides to come up with a better message.

@cel4 you can do a `assertRegExp`. And it _should_ break if something (e.g. the error message is changed), that's the point.

@jreback, sorry I am very busy at the moment. I am planning to continue the work here in a week or two.

I added a draft version how a test could look like. Would appreciate any comments.

`AttributeError: 'TestOrderedMerge' object has no attribute 'assertRegexpMatches'` - grr...  seems like `assertRegexpMatches` is not part of the `unittest` module in the `2.6` version...

@cel4 In the pandas `testing` module, there is a `assertRaisesRegexp` available for this -> `tm.assertRaisesRegexp()`

Hmh, tests are green. Yet I am not sure if I like this solution. Wouldn't it be better if we also check in`pd.concat` for empty sequences, so we can raise a even better (less general) error message about what went wrong exactly.

this needs to address my comments above
you can force materializing a generator and thrn just take the len

I propose "ValueError: No objects to concatenate" 

@sergeny, I like that suggestion. Unfortunately, I do not have much time at the moment and I totally forgot about that PR. I hope I can finish it soon. 

closing as stale, but if you would like to reopen and fixup pls do.

Should be ready for a first round of reviews.

looks reasonable. pls add a release note. and squash. (you can put in API changes)

@jreback, When travis is happy, this should be ready for review once again.

merged via https://github.com/pydata/pandas/commit/58db03eec9bd6d78976b844267c3c1a9f61a8eb3

thanks!
BUG: Fix for extraneous default cell format in xlsxwriter files.
State: closed

Fix for issue in the xlsxwriter engine where is adds a default
'General' format to cells if no other format is applied. This
isn't a bug, per se, but it prevents other row or column formatting.

closes #9167


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
@jmcnamara pls add a release note. This is a bug fix? user noticeable? (so an API change?) pls make the appropriate entry. thxs.

> add a release note. This is a bug fix? user noticeable? (so an API change?) pls make the appropriate entry. thxs.

I'm not sure if it needs a release note entry since it doesn't really affect pandas. It it mainly to allow users to format columns in the encapsulated xlsxwriter object. If you think that it needs an entry I'll add one. Should it go in `doc/source/whatsnew/v0.16.0.txt`?

it needs a note  - in v0.16.0 bug fix section

@jreback 
- Added release note.
- Used `_skip_if_no_openpyxl()` as noted above.
- Rebased and re-pushed.

merged via https://github.com/pydata/pandas/commit/07a573561422c83441b0e4023a73421d495d89f1

thanks!
TST: tests for GH5873
State: closed

closes https://github.com/pydata/pandas/issues/5873

code changes were done in https://github.com/pydata/pandas/pull/9101 and https://github.com/pydata/pandas/pull/9061. just adding tests to confirm bugs reported there are handled.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
merged via https://github.com/pydata/pandas/commit/28fc02a9eb285177143fb1b80e249fd75b6829b9