BrandonFors/pandas-github-issues huggingface dataset:
BUG:  Holiday(..) with both offset and observance raises
State: closed

Resolves #10217 

Doesn't fix the underlying issue yet, but at least raises on the unsupported behavior.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
pls add a release note (put it in the API section for 0.17.0). minor changes otherwise. pls ping when green on travis.

As requested. By the way, five points to whoever wrote the 'contributing to pandas' doc section.

thanks to 

cc @rockg who put most of that together!

cc @rockg

look good to you?

Looks good to me.  Let's keep the original issue open until that requested behavior is supported.

@bwillers Glad it was useful...hopefully removes the major hurdles to helping out.

merged via https://github.com/pydata/pandas/commit/d513986e75110b14e288753701e435660f3d115e

thanks!
Fix float formatting edge case
State: closed


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
I guess I changed this on purpose !

```
*** Processing results...
*** 
Invoked with :
--ncalls: 3
--repeats: 3


-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
read_csv_infer_datetime_format_ymd           |   1.9714 |   2.0163 |   0.9777 |
read_csv_comment2                            |  25.3403 |  25.4656 |   0.9951 |
read_csv_precise_converter                   |   1.5370 |   1.5430 |   0.9961 |
read_csv_roundtrip_converter                 |   2.3714 |   2.3573 |   1.0060 |
frame_to_csv2                                | 117.3016 | 115.5004 |   1.0156 |
read_csv_thou_vb                             |  19.5527 |  19.1456 |   1.0213 |
read_csv_default_converter                   |   1.6466 |   1.5797 |   1.0424 |
read_csv_infer_datetime_format_iso8601       |   1.9317 |   1.8443 |   1.0474 |
read_csv_infer_datetime_format_custom        |  10.3533 |   9.6080 |   1.0776 |
read_csv_standard                            |  11.9016 |  10.5267 |   1.1306 |
read_csv_vb                                  |  22.6383 |  18.4843 |   1.2247 |
packers_read_csv                             | 193.1477 | 156.2746 |   1.2360 |
frame_to_csv_date_formatting                 |  12.8030 |  10.2570 |   1.2482 |
frame_to_csv_mixed                           | 175.5013 | 138.3707 |   1.2683 |
read_csv_skiprows                            |  16.4720 |  12.2391 |   1.3459 |
packers_write_csv                            | 1086.9323 | 621.1909 |   1.7498 |
write_csv_standard                           |  45.0310 |  24.5957 |   1.8309 |
frame_to_csv                                 | 126.7430 |  54.7020 |   2.3170 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [4c4497c] : Fix float formatting precision
Base   [1d87174] : BUG: plot doesnt default to matplotlib axes.grid setting (#9792)
```

Alright, I'll take a look and see if I can get the perf up to snuff with master

@cpcloud any luck with the perf fix on this?

@cpcloud status?

@cpcloud closing, but reopen if still interested
Update plotting.py
State: closed

Should fix https://github.com/pydata/pandas/issues/10214#issuecomment-106091734


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Thanks @scls19fr !

Can you add:
- test cases (pls see `test_graphics.py`)
- release note for the bug fix (`doc/source/whatsnew`)

Sorry @sinhrks but I feel:

![engrenage](https://cloud.githubusercontent.com/assets/109167/7885929/0272a112-0627-11e5-9696-e18f572e2045.jpg)

In `pandas/doc/source/whatsnew/v0.17` there's a bunch of one-line notes describing bug fixes. Add one like that.

Tests for this are in `pandas/tests/test_graphics.py` The idea is to make a test that fails without your change, but works with your change. There's a bunch of helpful info here: https://github.com/pydata/pandas/wiki/

Let us know what you're stuck on, and we can walk you through it. Most of us are in the [chat room](https://gitter.im/pydata/pandas) if you want quicker feedback on git stuff.

Superseded by https://github.com/pydata/pandas/pull/10233
add try / except to address issue #10154 to_datetime, Inconsistent be…
State: closed

closes #10154 to_datetime, Inconsistent behavior with invalid dates.

results are now

```
In [2]: pd.to_datetime('2015-02-29', format="%Y-%m-%d", coerce=True)
Out[2]: NaT

In [3]: pd.to_datetime('2015-03-32', format="%Y-%m-%d", coerce=True)
Out[3]: NaT

In [4]: pd.to_datetime('2015-02-32', format="%Y-%m-%d", coerce=True)
Out[4]: NaT

In [5]: pd.to_datetime('2015-04-31', format="%Y-%m-%d", coerce=True)
Out[5]: NaT
```


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
needs tests

can you rebase

On Fri, Jun 26, 2015 at 5:39 PM, jreback notifications@github.com wrote:

> can you rebase

I will work on that this weekend.
ENH: less verbose merging on index
State: closed

This is a proposal for a simple API enhancement that would make merging on indices much more convenient. I've put in an initial commit to show the change needed in the library. Here's the old way of merging on indices:

``` python
In [1]: from pandas import Index, DataFrame, merge

In [2]: df1 = DataFrame({'data1':[0, 1, 2]})

In [3]: df2 = DataFrame({'data2':[3, 4, 5]})

In [4]: merge(df1, df2, left_index=True, right_index=True)
Out[4]: 
   data1  data2
0      0      3
1      1      4
2      2      5
```

Here is the new behavior enabled by this PR:

``` python
In [5]: merge(df1, df2, on=Index)
Out[5]: 
   data1  data2
0      0      3
1      1      4
2      2      5
```

This is a bit hackish, I admit, but I find myself wanting to do this often and the alternatives (using `left_index` and `right_index`, or using `concat([df1, df2], axis='cols')`) are inconvenient and less clear to the reader of the code. That said, I completely understand if folks think this is not a good fit for the pandas API.

Any thoughts on this idea?


Labels:
name: Docs

Comments:
I agree that the `left_index=True, right_index=True` is rather verbose, and some improvement of the API is welcome here. Thanks for initiating this!

But I also agree this (using `on=pd.Index`) is bit hacky, or at least, not really consistent with how such things are done in the rest of pandas.

So maybe it is also worth looking at what could be alternatives, before deciding on the API?
- a new keyword, e.g. `on_index=True`
- promote `df1.join(df2)`, which joins on the index by default
- add a `pd.join(df1, df2)` that does the same as `DataFrame.join` (similar to pd.merge vs DataFrame.merge). So a `pd.merge` alternative that has different defaults
- ...

Other ideas?

Thanks! I didn't realize that `df.join()` will join on index by default. Perhaps just emphasizing/promoting that more would be sufficient?

@jakevdp would love to have an addition to the docs to emphasize this. 

FWIW, I have always been confused about the distinction between `pd.merge` and `pd.join`.

closing as this is just diff between `merge` and defaults in `join`. suppose docs could be added.
DOC: minor doc fix for Series.append; indices can overlap
State: closed

Old docstring seemed to be incorrect; overlapping indices are supported e.g.

``` python
>>> import pandas as pd
>>> x = pd.Series(['A', 'B', 'C'])
>>> y = pd.Series(['D', 'E', 'F'])
>>> x.append(y)
0    A
1    B
2    C
0    D
1    E
2    F
dtype: object
```


Labels:
name: Docs

Comments:
This would raise if `verify_integrity=True` though, seems like what that was meant for. 

```
In [6]: x.append(y, verify_integrity=True)
ValueError: Indexes have overlapping values: [0, 1, 2]
```

Perhaps a good idea to do this?

```
 The indexes must not overlap if verify_integrity is True
```

The documentation of `verify_integrity` already says this pretty clearly. I'm not sure the information needs to be repeated in the general description. For what it's worth, [`DataFrame.append`](https://github.com/pydata/pandas/blob/master/pandas/core/frame.py#L3886) and [`pd.concat`](https://github.com/pydata/pandas/blob/master/pandas/tools/merge.py#L700) both have the same behavior, and don't mention anything about duplicate indices in the general description.

I think this looks good!

@jakevdp ah I see, yeah that makes sense

@jakevdp Thanks!
BUG: Raise TypeError only if key DataFrame is not empty #10126
State: closed

Proposed fix for #10126


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
change this to close issue #10126 instead

@jreback Requested changes made and squashed, changed title to reference #10126

Using `is_bool_dtype` results in a couple of extra test failures- investigating.

``` python
======================================================================
FAIL: test_setitem_boolean (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\dev\code\opensource\pandas-rekcahpassyla\pandas\tests\test_frame.py", line 621, in test_setitem_boolean
    df[df * 0] = 2
  File "c:\dev\code\opensource\pandas-rekcahpassyla\pandas\util\testing.py", line 1640, in __exit__
    raise AssertionError("{0} not raised.".format(name))
AssertionError: TypeError not raised.

======================================================================
FAIL: test_frame_getitem_setitem_boolean (pandas.tests.test_multilevel.TestMultiLevel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\dev\code\opensource\pandas-rekcahpassyla\pandas\tests\test_multilevel.py", line 354, in test_frame_getitem_setitem_boolean
    df[df * 0] = 2
  File "c:\dev\code\opensource\pandas-rekcahpassyla\pandas\util\testing.py", line 1640, in __exit__
    raise AssertionError("{0} not raised.".format(name))
AssertionError: TypeError not raised.
```

Fixed my obvious silly mistake. (Doh)

ok, looks good. pls add a release mentioning the original issue.

Rebased and added release note to 0.16.2, in the bug fix section. 

thanks!

Thank you! 

Since this is merged in master, all travis builds seem to fail (although the tests for the PR itself were green):

```
FAIL: test_sum (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_series.py", line 2319, in test_sum
    self._check_stat_op('sum', np.sum)
  File "/home/travis/build/pydata/pandas/pandas/tests/test_series.py", line 2683, in _check_stat_op
    testit()
  File "/home/travis/build/pydata/pandas/pandas/tests/test_series.py", line 2656, in testit
    self.assertTrue(np.isnan(f(allna)))
AssertionError: False is not true
```

Ah, caused by bottleneck upgrade: #10270
BUG: Check for size != 0 before trying to insert #10193
State: closed

Proposed fix for #10193


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
see my comments on the actual issue. This needs to test that it is:
- hitting the `SetttingWIthCopy` path
- affecting the referenced Series (e.g. you have a view in this case).

I had another go- I looked at how `DataFrame` handled the setting of `is_copy` and then raising `SettingWithCopyError` or `SettingWithCopyWarning`, and tried to replicate that logic in `Series`. 

Added tests for the requisite error / warning being raised, and that the original series is unchanged. Had to change a couple of other tests to change the `chained_assignment` option to `None` and back again. 

I haven't squashed commits as it's still WIP. Will look again to see how I can make it more consistent with existing functionality, but here's something for now. 

Noted the build failures. Investigating. 

Requested changes from review have been made, also replaced other existing instances of `pd.set_option('chained_assignment', ...)` with context manager, to be consistent. 

I have to think about this again. This is a 'feature' of numpy. In that if you set something on a view of something else it works, and changes the original, as that is the point. However, in pandas, objects are pure and changing one shouldn't impact another one, so in effect, even though its supported, the view machinery is effectively frowned upon.

@shoyer @jorisvandenbossche thoughts?

@jreback can you clarify the specific behavior you're not sure about?

Well its just general view semantics for Series. These almost always apply because these are a single dtyped, while for frames depends on whether its multi-dtype and what you are setting. I am just thinking that this might be confusing.

```
In [1]: s = Series([1,2,3])

In [2]: s
Out[2]: 
0    1
1    2
2    3
dtype: int64

In [3]: s2 = s[1:]

In [4]: s2
Out[4]: 
1    2
2    3
dtype: int64

In [5]: s2.iloc[0] = 5

In [6]: s2
Out[6]: 
1    5
2    3
dtype: int64

In [7]: s
Out[7]: 
0    1
1    5
2    3
dtype: int64
```

I agree, views are a bit confusing, but they're also a fundamental part of the NumPy ecosystem -- and they make a lot of efficient things possible.

I would definitely prefer not to add more SettingWithCopy warnings. To quote @mwaskom from [Twitter](https://twitter.com/michaelwaskom/status/601405003186577408):

> IMO warnings are almost never a good way to communicate usage notes to users for exactly this reason. They annoy people who know what they are doing, and they rarely provide enough context for those who don't.

For my own edification; is the issue here that setting a value on a Series which is a view should always pass, because "in pandas, objects are pure and changing one shouldn't impact another one", or that it should always fail? 

closing in favor of #10379
BUG: Adding empty dataframes should result in empty blocks #10181
State: closed

Fixes #10181 

I added a couple of lines to the dataframe test `test_operators` to assert that  adding two empty dataframes returns a dataframe that is equal (both using `assert_frame_equal` as well as `.equals`) to an empty dataframe.

I couldn't get vbench to install (I am running on windows) - the error was:

```
c:\dev\code\opensource\pandas-rekcahpassyla>pip install git+https://github.com/pydata/vbench
You are using pip version 6.0.8, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting git+https://github.com/pydata/vbench
  Cloning https://github.com/pydata/vbench to c:\users\redacted\appdata\local\temp\pip-ruiahg-build
fatal: unable to connect to github.com:
github.com[0: 192.30.252.129]: errno=No error

Clone of 'git://github.com/yarikoptic/vbenchtest.git' into submodule path 'vbench/tests/vbenchtest/vbenchtest' failed
  Complete output from command C:\dev\bin\git\cmd\git.exe submodule update --init --recursive -q:

  ----------------------------------------
  Command "C:\dev\bin\git\cmd\git.exe submodule update --init --recursive -q" failed with error code 1 in c:\users\redacted\appdata\local\temp\pip-ruiahg-build
```

The test suite failed for me in master with 1 failure. The same failure, but no others, were observed when running in my branch. 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
Added more tests and comment as suggested

@rekcahpassyla ok this looks reasonable. pls add a release note to whatsnew/v0.17.0 (you can put in other API changes section).

pls squash as well.

Added release note and squashed.

I note the build failure, will have a look. 

I have verified that my latest commit passes the `pandas.stats` tests (the cause of the build failure). However I haven't squashed it, as I don't know if it would be considered a separate bug (searching the issues list didn't reveal anything). If it is a separate bug, I'm happy to open another issue and cherrypick this commit into another PR. 

Raised  https://github.com/pydata/pandas/issues/10195 and proposed https://github.com/pydata/pandas/pull/10196 to fix the issue causing https://travis-ci.org/pydata/pandas/builds/63544612 to fail. 

pls rebase this (and move the release note to 0.16.2)

I think these are some old travis failures. can you rebase on master. ping when green.

@jreback - It is green

thanks!
DOC: Add Index.difference to API doc
State: closed

Add `Index.difference` to api doc rather than deprecated `Index.diff`  (#8226)

And is it OK to remove `Index.diff` alias because #6581 lists #8227 (which closes #8226)?


Labels:
name: Docs

Comments:
yep no problem with changing this in the API docs (but move as @jorisvandenbossche mentions)

The deprecation of `Index.diff` is actually not explicitely mentioned in #6581, we should add that. Also, we should clean up that list for 0.17, as not all things listed there should already be removed in 0.17

@jorisvandenbossche yep, that list really should be for 'next-major-release' then separate section for 0.17.0

I think it is natural under the same category as `intersection` and `union`. How about moving `difference` and `sym_diff` to "Combining / joining / merging" category, renaming it to "Combining / joining / set operations"?

yes, sounds good to me

agreed, making a new sub-section for setops would be gr8.

OK, did the suggested change. Pls check.
FIX printing index with display.max_seq_items=None (GH10182)
State: closed

Fixes #10182 


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@jreback two reasons for the failures (both because I used `repeat`):
- CategoricalIndex -> `repeat` does not work, as Categorical has no repeat method -> do I add this?
- PeriodIndex -> `repeat` removes the `freq` -> this seems like a bug? (or not specifically implemented/checked for PeriodIndex)

I think if you remove `tseries/base/DatetimeIndexOpsMix/repeat` the `PeriodIndex` case will be fixed (as `core/index.py/Index/repeat` should work; this correctly propogates the meta-data). Obviously not tested though.

I think `CategoricalIndex` might need its own repeat, though you _might_ be able to 'combine' it with `MultiIndex.repeat` as these are pretty similar. (You can create a mixin, maybe `MultiIndexMixin`) or something to hold these (and a couple more methods IIRC).

For the CategoricalIndex, why not adding `Categorical.repeat`? This seems rather straightforward at first sight, as repeating the codes should do? And then no logic on the Index itself has to be changed.

@jorisvandenbossche that's what I said, add for `CategoricalIndex.repeat` (you can combine with `MultiIndex.repeat` if you want to (side issue)

I mean `Categorical.repeat`, not `CategoricalIndex.repeat`

@jorisvandenbossche oh, yes, for sure do that!

@jreback can you have a look?

this looked reasonable

@jreback yes, coming back to this. I think your main comment here is that the tests can be expanded by looping over the `indices`. However, as I am rather busy at work this week, is it OK to leave that for another PR? Besides the whatsnew notice I think this is good to go.

@jorisvandenbossche sure. Always like expanded tests, but otherwise ok. merge when ready.

ok, will do this evening
BUG: concat on axis=0 with categorical (GH10177)
State: closed

Contains a proposed fix for #10177


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
@jreback: I implemented your suggested changes and it seems that it doesn't break anything else, at least according to the unit tests.

I rebased the changes against latest master. The Travis CI error is due to a timeout when creating the build environment, not sure how I can resolve it.

pls rebase and squash.

@jreback: I updated my merge request based on your feedback.

@sebp thanks!
BUG: mean overflows for integer dtypes (fixes #10155)
State: closed

closes #10155 


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
There are technically two changes now:

### old behavior
1. For `int` dtypes, `mean()` first sums up values in `int` and converts to `float`, therefore it can potential have int overflow
2. For `float` dtypes, `mean()` sums up values in `float64` and returns the result in `float64`, even if the input dtype is a lower precision dtype such as `float32` 

### new behavior
1. For `int` dtypes, `mean()` sums up values in `float64` and returns the result as `float64`
2. For `float` dtypes, `mean()` sums up values using the same input `float` dtype and returns the result in the same input dtype. Namely, `float32` input will have both the computation and return value in `float32`

Both points in the old behavior are not consistent with `numpy`, and both points in the new behavior are. 

LGTM... just needs a release note

ok - in another issue should audit the rest of the nan funcs for overflow / dtype preserving effects
- lets just put in place some tests

So I was actually quite puzzled by why the unit tests wouldn't pass for some of the virtual environments. And I think I tracked down the problem ... seems like a bug in `numpy < 1.9.0`. For python 3.4 this is not a problem because `numpy >= 1.9.0` is required, but for other environments with older numpy versions this can fail

```
(python2):~/code/github/pandas$ nosetests -s pandas/tests/test_nanops.py 
............S.S....S....F....SS...
======================================================================
FAIL: test_nanmean_overflow (pandas.tests.test_nanops.TestnanopsDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mortada/code/github/pandas/pandas/tests/test_nanops.py", line 336, in test_nanmean_overflow
    self.assertEqual(result, a)
AssertionError: 20150515061816464.0 != 20150515061816532
```

I verified that `np.mean()` produces this wrong value for older versions prior to 1.9.0.

Ok I've updated this with a numpy version check, travis should pass now. Also added a release note. 

@jreback I added an optional `dtype` parameter to `_get_counts()` as you suggested 

@jreback @shoyer this should be ready for review. 

I'll create another issue/PR to audit the rest of the nan funcs for overflow and returned dtypes

Indeed, I think this is good to go... I'll wait a little bit and then merge.

yep, lgtm

thanks @mortada !

cool thanks guys!