BrandonFors/pandas-github-issues huggingface dataset:
PERF: Improve replace perf
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

When `.replace` is called with `dict`, replacements are done per value. Current impl try to soft convert the dtype in every replacement, but it is enough to be done in the final replacement.
#### Bench

```
-  712.83ms   355.93ms      0.50  replace.replace_convert.time_replace_series_timestamp
-     1.50s   698.21ms      0.46  replace.replace_convert.time_replace_frame_timestamp
-     3.12s   690.48ms      0.22  replace.replace_convert.time_replace_frame_timedelta
-     1.69s   354.83ms      0.21  replace.replace_convert.time_replace_series_timedelta
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
notice that `BoolBlock.replace` doesn't have `convert` arg at all! so not tested / working. can you give a check.

Ah, found some `.replace()` conversion looks buggy. I'll submit a separate issue as the fix not looks straightforward.

See #12747. Change this PR title to WIP until #12747 is fixed.

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/12745?src=pr) is 85.27% (diff: 90.00%)
> Merging [#12745](https://codecov.io/gh/pandas-dev/pandas/pull/12745?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #12745   diff @@
==========================================
  Files           144        144          
  Lines         50915      50921     +6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43419      43423     +4   
- Misses         7496       7498     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [dfeae39...ffc59b0](https://codecov.io/gh/pandas-dev/pandas/compare/dfeae396c832358e0a2f64e67bd48ca83a9c4976...ffc59b04d2328e3bf7059bec615f99e8dcfbf8a4?src=pr)
can you rebase / update?

lgtm. can we merge this independent of the other PR's? (so it could go in 0.19.2)
It is independent. #12780 fixes some existing bugs separately.
thanks!
BUG: SparseSeries.shape ignores fill_value
State: closed

- [x] closes #10452
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Based on existing tests / text search, no other metohd depends on current `SparseSeries.shape` (buggy) behavior.


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
thanks!
COMPAT: compat with released numpy 1.11 for IndexError -> TypeError
State: closed

was a revert of # https://github.com/numpy/numpy/pull/6271

closes #12729
closes #12792 


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
waiting for `numpy==1.11.0` to be live in conda for proper testing, as we don't want to add a pip only build (as we are already testing from numpy master, which worked, except for this case where the version determination changed after we put our fix in).
Fix pandas.Timedelta range
State: closed

- [x] closes #12727
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

_Problem_
Pandas Timedelta derives from `datetime.timedelta` and increase
the resolution of the timedeltas. As such the Pandas.Timedelta
object can only have a smaller range of values.

_Solution_
This change modifies the properties that report
the range and resolution to reflect Pandas capabilities.

**Reference**
https://github.com/python/cpython/blob/8d1d7e6816753248768e4cc1c0370221814e9cf1/Lib/datetime.py#L651-L654


Labels:
name: Timedelta
description: Timedelta data type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
This concerns the 2nd commit, in this PR

@jreback, can you confirm that some of the tests [at](https://github.com/pydata/pandas/blob/b6f0292cd17b743528390a9e9efe92db96f8c19e/pandas/tseries/tests/test_timedeltas.py#L1678) rely on buggy output.

Specifically, that `result` (slicing in the snippet below) erroneously includes the end day.

``` python
rng = timedelta_range('1 day 10:11:12', freq='h', periods=500)
s = Series(np.arange(len(rng)), index=rng)

result = s['5 day':'6 day']
expected = s.iloc[86:110]
assert_series_equal(result, expected)
```

If this is a second bug fix, then I will address the changes documentation. Maybe there is even a better way other than peaking at the indices and creating the `expected` slices.

pls add a whatsnew note, point to the new docs (with a `:ref:`). Pls also move the `datetime` limits section to `timeseries.rst` as well.

thanks.

thanks @has2k1
Omit tests folders from coverage
State: closed

xref #12634 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
can you tell me what change this has on the coverage numbers? e.g. going from xxx with tests to yyy w/o?

Sure. current master 88.68%
https://codecov.io/github/pydata/pandas
With tests excluded 83.63% (subject to change as the build is red ...)
https://codecov.io/github/pydata/pandas?pr=12721
https://codecov.io/github/pydata/pandas?ref=af4e9df23498a6e4ed1980a347aad12a1e4be2a0 (from above in this PR)

@gliptak thanks!
Convert badge list into Markdown table in README.md
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Also correct some badge links


Labels:
name: Docs

Comments:
@gliptak can you past a screen shot of what this looks like.

@jreback  see https://github.com/gliptak/pandas/blob/patch-1/README.md

![screenshot from 2016-03-29 13 19 18](https://cloud.githubusercontent.com/assets/1217238/14122481/e0e173be-f5b0-11e5-8f3e-6ba945f81e3f.png)

There seems to be a visual glitch at the top of the table

oh, so this is equiv of the html that we had before (which I ripped off).

what is the glitch?

The "glitch" is a way to have an empty table header in Markdown. Comments on improvements are welcome :)

If we can't have an empty table header in markdown, I would stick with HTML.

How about a title (Description/Badge)? Or just have them in a row like most projects do:
https://github.com/xmlrunner/unittest-xml-reporting
https://github.com/kennethreitz/requests

I think the rows are bad looking
much prefer he table

let's leave the existing. of course if a markdown version is better (but preserves the structure), then pls re-open.
Raising TypeError on invalid window when using .rolling #12669
State: closed

- [x] closes #12669
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
competing PR is #12714 

can you rebase/update according to comments

can you rebase / update

superseded by #13002
COMPAT: make read_excel accept path objects for the filepath (#12655)
State: closed

- [x] closes #12655
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Fairly simple fix overall, since `get_filepath_or_buffer` is already set up to just convert the path objs to string, we can do that first and then just let the existing logic handle it.

Since the libraries that contain the path objects aren't hard dependencies, I'm not sure how to typecheck against them without first using the `_IS_LIBRARY_INSTALLED` checks and then importing if they're installed. Let me know if this should be done differently.


Labels:
name: IO Excel
description: read_excel, to_excel
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
OK, what if we just use the `_stringify_path()` function that already exists to avoid the ugly imports? I'm pretty sure `ExcelFile` needs to do slightly different things with URLs than what is done in `get_filepath_or_buffer`, in terms of getting the data as string vs bytes.

@jreback Squashed and tests are passing, do I need to worry about the coverage checker? Haven't seen that before.

@onesandzeroes I think its complaining (the cov checker) about not testing the `is_url` calls. Can you review the tests and make sure we are testing pulling from url and s3 ok? (I think we should be). ping if ok.

@jreback yeah there are tests for both http and file url's, and for s3 as well, those paths should be covered.

@onesandzeroes thanks!
Allow float window when using .rolling
State: closed

- [ ] closes #12669
- [ ] tests added
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

https://github.com/pydata/pandas/issues/12669


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
I just noted in the associated issue that that I think it would be better to raise than coerce integers.

It could be OK to accept floats if `int(window) == window`, but `window=2.5` should definitely raise an error.

I agree we should just raise on non-int. I think was an oversight in the initial impl. yes its safe if `int(x) == x`, but more correct to raise `TypeError` as @shoyer suggests.

competing PR is #12718 

@gliptak want to update (also see my comments on the competeting PR).
Fix #12564 for Categorical: consistent result if comparing as DataFrame
State: closed

- [x] closes #12564 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
pls add a whatsnew note in Bug Fixes as well (for 0.18.1)

ok, I'll add a whatsnew note by tomorrow.

Thanks for your advice, @jreback.
Is it OK? and do I need to rebase for resolving conflicts?

@ningchi if you could that'd be great. It's probably just a conflict in the whatsnew file. And if your comfortable you can squash your commits down to a nice single message too.

yes, pls rebase

many thanks! I have rebased these commits.

thanks @ningchi
Fix #12695 : RangeIndex generated with pd.concat with ignore_index as True
State: closed

- [x] closes #12695
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Replaced generated indexes with RangeIndex, test still needed


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
tests! always start with the tests to make sure it fails first!

- needs some tests (use the one from the original issue)
- whatsnew entry (bug fix)

Thanks for the PR:) But I've included the fix in #12846 to refactor existing code. Pls lmk if anything.
Fix : 9784 Added the note for display.max_colwidth in dsintro.rst
State: closed

- [x] closes #9784
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
ok sir, i will add an example with Long names or strings that will make it more clear. 

sir, any reviews

filename changed to generic

thanks!
BUG: .describe lost CategoricalIndex info
State: closed

- [x] closes #11558
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
#11558 is partially fixed by #12531, this PR let `.describe()` to preserve `CategoricalIndex` info. And added explicit test cases derived from #11558.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
@sinhrks yeah that looks right. Pls have a quick look thru the index classes and confirm if that's really what we are doing for the rest. You _might_ even be able to lift _some_ of the stuff off of `CategoricalIndex._simple_new` to `Categorical_index._shallow_copy` (and then change things that use `_simple_new`. It might not have been exactly clean to start.

@jreback Current `CategoricalIndex._simple_new` follows the above definition (not rely on metadata at all).

https://github.com/pydata/pandas/blob/master/pandas/indexes/category.py#L126

thanks @sinhrks always nice PR's!
Fixed #12651: providing links to the  .expanding in .cum*
State: closed

- [ x] closes #12561
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
#12561 : wrote the `.expanding` function int the `basics.rst`


Labels:
name: Docs

Comments:
xref #12673. Should close #12651 (not #12561).

sectional for the expanding provided.

move these commits to #12673
Fixed #12651: providing links to the .cum* in .expanding
State: closed

- [x] closes #12651
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
  #12651
  
  commit for the .cum\* in .expanding


Labels:
name: Docs

Comments:
Should close #12651 (not #12561).

I think it is better to link each other via docstring as described in #12561, or use section links ( `..sealso::` ). It's unclear what the updated part tries to explain.

sectional links provided.

pls reopen if you like to update according to comments
Fixed #12661: more clarification in the where statement
State: closed

- [x] closes #12661
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Improvement in the documentation w.r.t to issue #12661


Labels:
name: Docs

Comments:
Personally, I feel explanation of `.where()` is inappropriate here. Because the section focuses on `df[df.A > 0]` and `df[df > 0]`

Maybe following sentence should be fixed, or pls add an explanation if you add `.where()` example.

> A `where` operation for getting.

ok sir, i will add brief explanation to `where` 

I agree with @sinhrks, I don't think the explanation of `where` is needed here. It is a section about boolean indexing (which does a 'where' operation), not about the `where` method. And I don't think it is needed to mention the `where` method in an 10min intro (there is already a section on `where` in the docs: http://pandas.pydata.org/pandas-docs/stable/indexing.html#the-where-method-and-masking).

And sorry that this was not really clear from the issue #12661. 

I would just remove the ```` around 'where' to make it clear that it is not a method.

> I would just remove the `` around 'where' to make it clear that it is not a method.

so, I just remove the `` from where 
and also it is given in detail in the indexing section:
 http://pandas.pydata.org/pandas-docs/stable/indexing.html#the-where-method-and-masking

so, any further work to be done? 

closing, but if updated pls reopen
ENH: xz compression in to_csv() resolves #11852
State: closed

- [x] closes #11852
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
can do this on top of #12175 as that added compression inference and reorged the testing

Updated docs to include reading, using PY3 / PY2 instead of except, will rebase when #12175 gets merged.

ok #12175 was merged, pls rebase

also note #12688 

interesting this only appears in 2 locations (and not the 3rd). any idea?

@jreback Rebased, will look into #12688 

@jreback For now I will also add it to `parsers.py`

@jreback Updated

minor doc fix

@jreback fixed typo

ok, lgtm. ping when green.

@jreback green

@terfn thanks!

I refactored to remove some duplicate code, but looks really good!.
Updated docstring for copy() method
State: closed

- [x] closes #12663 

Added note that a copy with `deep=True` does not copy the data of objects contained in underlying arrays with `dtype=object`.


Labels:
name: Docs

Comments:
Actually I think `deep=False` doesn't copy the data at all:

```
d = pd.DataFrame(rand(3,3))
c = d.copy(deep=False)
c.loc[0,0] = -9
print d
```

Seems like `deep=False` is a view of the data and uses the exact same objects for index and columns,
whereas `deep=True` is a copy of the data and a copy of the index and columns (except for dtype=object)

I tried to be more explicit in this version. I hope this is helpful.

thanks @agartland
BUG: fixed SAS7BDATReader._get_properties
State: closed

this fixes a crasher I had in one of my files using SAS7BDATReader.

cc: @kshedden 

self._path_or_buf.read is an obvious mistake

also added + total_align to _os_name_offset and _os_make_offset as they are
present in the original code from Jared Hobbs:
https://bitbucket.org/jaredhobbs/sas7bdat/src/da1faa90d0b15c2c97a2a8eb86c91c58081bdd86/sas7bdat.py?fileviewer=file-view-default#sas7bdat.py-1450


Labels:
name: Bug
name: IO SAS
description: SAS: read_sas

Comments:
can you add a test that hits this path

See #12659 test file. It fails too.
https://github.com/pydata/pandas/blob/8857008178eb90b59ca67874507ef860b3faf88f/pandas/io/tests/sas/data/test17.sas7bdat

Do you want me to include that test file in this PR even if the file is **not** readable even with this PR (due to #12659)?

prob easiest to have a single PR with the test files.

btw, this fixes issue #12647 

I've uploaded a test file here: https://github.com/raderaj/pandas_SAS_snippets/blob/master/productsales.sas7bdat 

closed by #12656
BUG: pivot_table dropna=False drops columns/index names
State: closed

- [X] closes #12133 
- [X] closes #12642, closes #12327 (replace PR)
- [X] tests added / passed
- [X] passes `git diff upstream/master | flake8 --diff`
- [X] whatsnew entry

Bug fixed from pull request #12327.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
@jreback Thanks for the comments. But I don't think this is the correct way to cherry pick commits..?? I'll learn some github basics and fix this thing within next two days.

@OXPHOS it looks ok. you prob want to squash commits down though to make it easier to work with (e.g. squash yours to a single one, his the same).

Cool. Got it. Thanks!

On Thu, Mar 17, 2016 at 12:19 PM, Jeff Reback notifications@github.com
wrote:

> @OXPHOS https://github.com/OXPHOS it looks ok. you prob want to squash
> commits down though to make it easier to work with (e.g. squash yours to a
> single one, his the same).
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/12650#issuecomment-197956057

@jreback ready to merge : )

thanks @OXPHOS
BUG: crosstab with margins=True and dropna=False 
State: closed

- [+] closes #12642 
- [+] tests added / passed
- [+] passes `git diff upstream/master | flake8 --diff`
- [+] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> Powered by [Codecov](https://codecov.io). Updated on successful CI builds.

I'll wait for #12327 

make sure you rebase on master (to pick up the merges from the other issue). 

you can submit a separate PR that does #12327 if you want (just incorporate those commits) and fixup as I have suggested.

- Rebased.
- #12327 already solved the problem.
- The test file I generated may be useful
- But I don't know how to include commits from #12327 .  I assume I can't just copy his final codes. I tried forking from him but it didn't work like that.

ideally you would clone his branch
then cherry pick commits you want onto yours
MAINT: PEP8 + spelling fixes in testing.py
State: closed


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
## [Current coverage](https://codecov.io/github/pydata/pandas?ref=7f8a90b7cb86d5f2d49f6cdb0c7ae9a23eda5ad8) is `88.64%`

> Merging **#12637** into **master** will not affect coverage as of [`7f8a90b`](https://codecov.io/github/pydata/pandas/commit/7f8a90b7cb86d5f2d49f6cdb0c7ae9a23eda5ad8)

``` diff
@@            master   #12637   diff @@
=======================================
  Files          275      275       
  Stmts       133688   133688       
  Branches         0        0       
  Methods          0        0       
=======================================
+ Hit         118511   118512     +1
  Partial          0        0       
+ Missed       15177    15176     -1
```

> Review entire [Coverage Diff](https://codecov.io/github/pydata/pandas/compare/f71537ab2561ab5727008095e9685966619fa7b9...7f8a90b7cb86d5f2d49f6cdb0c7ae9a23eda5ad8) as of [`7f8a90b`](https://codecov.io/github/pydata/pandas/commit/7f8a90b7cb86d5f2d49f6cdb0c7ae9a23eda5ad8)
> 
> Powered by [Codecov](https://codecov.io). Updated on successful CI builds.

theres a flake directive at the top of the file, need to remove it

Removed the directive as well as all of the `flake8` violations, and Travis gives the green light.  Should be good to merge.

thanks!
Allow dtype='category' or dtype=None for Series with Categorical
State: closed

Allows user to pass dtype='category' when creating a Series with Categorical, but produces a ValueError is any non-categorical dtype is passed.
- [x] closes #12574
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
Just the one comment, +1 otherwise (assuming Travis passes). Thanks!

## [Current coverage](https://codecov.io/github/pydata/pandas?ref=724c3091bff822f95aedd786a44be9e85c242511) is `88.64%`

> Merging **#12636** into **master** will not affect coverage as of [`724c309`](https://codecov.io/github/pydata/pandas/commit/724c3091bff822f95aedd786a44be9e85c242511)

``` diff
@@            master   #12636   diff @@
=======================================
  Files          275      275       
  Stmts       133688   133695     +7
  Branches         0        0       
  Methods          0        0       
=======================================
+ Hit         118511   118520     +9
  Partial          0        0       
+ Missed       15177    15175     -2
```

> Review entire [Coverage Diff](https://codecov.io/github/pydata/pandas/compare/f71537ab2561ab5727008095e9685966619fa7b9...724c3091bff822f95aedd786a44be9e85c242511) as of [`724c309`](https://codecov.io/github/pydata/pandas/commit/724c3091bff822f95aedd786a44be9e85c242511)
> 
> Powered by [Codecov](https://codecov.io). Updated on successful CI builds.

thanks!
BUG: Concat with tz-aware and timedelta raises AttributeError
State: closed

- [x] closes #12620
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

@jreback 's comment on #12620:

> should be handled at a slightly higher level.

I added number of `dtype` check logic in `tseries/common/_concat_compat`. But it should work on `base/common/_concat_compat`. Lmk which is preferable.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timedelta
description: Timedelta data type
name: Timezones
description: Timezone data dtype

Comments:
> Powered by [Codecov](https://codecov.io). Updated on successful CI builds.

thanks @sinhrks
CI: add in support for coverage via Codecov
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
so there is a nice browser extension to show the coverage on github: https://github.com/codecov/browser-extension#codecov-extension

will have active coverage once this PR builds. I think I have it set for PR's to show coverage updates (where the travis/ci line is).

cc @jorisvandenbossche @shoyer @TomAugspurger @sinhrks @wesm 

you can look at/modify things here: https://codecov.io/github/pydata/pandas/commits