BrandonFors/pandas-github-issues huggingface dataset:
Style display format
State: closed

Closes https://github.com/pydata/pandas/issues/11692
Closes https://github.com/pydata/pandas/issues/12134
Closes https://github.com/pydata/pandas/issues/12125

This adds a `.format` method to Styler for formatting the display value
(the actual text) of each scalar value.

In the processes of cleaning up the template, I close #12134 (spurious 0)
and #12125 (KeyError from using `iloc` improperly)


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Still a WIP for now, need to add handling for `subset` for some cases. Wanted to get this up since it's come up a couple times.

Ok, ready for review whenever someone gets a chance. These changes should be entirely backwards compatible.

Thanks Jeff.

I fixed everything except the `select_dtypes` enhancement. Do that as a followup since presumably it would be for all the style methods, not just `format`?

did u update the docs with an example for this?

Yeah, I've uploaded a [gist](http://nbviewer.jupyter.org/gist/TomAugspurger/071685a20c0a779e1ba4#Finer-Control:-Display-Values). I'm not sure why the diff was so badly mangled.

I looked into [nbsphinx](http://nbsphinx.readthedocs.org/en/0.2.3/) a bit and it looks promising. We'll need to tweak a few things to get it working with our site. I plan on doing that sometime.

All green here.

I need to figure out why the doc build [failed here](https://travis-ci.org/pydata/pandas/jobs/107922605). Haven't been able to reproduce it locally yet.

lgtm. merge when ready.

I'll get all my PRs finished up tonight. If I can't figure out why this is causing the sphinx build to fail I'll drop the changes to the notebook and do that in a followup PR.

still bombing sphinx?

Surprisingly, yes. Even though I removed my changes to the docs. I've got one more thing to try out this morning. 

> On Feb 11, 2016, at 21:14, Jeff Reback notifications@github.com wrote:
> 
> still bombing sphinx?
> 
> —
> Reply to this email directly or view it on GitHub.

@TomAugspurger Looking at the doc build log, it gives an error with "ImportError: No module named abc" (for the `from collections.abc import MutableMapping` that you added). That can possibly explain also all the other problems with sphinx. 
I get that same error message if I just try the import, but strange that the other tests do not fail. (a python 2 vs 3 issue?)

Thanks Joris! Hopefully that was it. It is strange that the tests didn't fail on py27. We do skip on the `27_slow_nnet_LOCALE` job, since we had a strange interaction between Jinja and matplotlib. Not sure why the other ones didn't fail.

Yeah, very strange .. I checked and they are certainly not skipped on some of the python 2.7 builds

@jreback passed without breaking sphinx, if you want to merge.

gr8!

thanks!

and of course can just join the travis queue :)

thanks @TomAugspurger 

I realized after I pressed Y that we don't have all of the issues listed in the whatsnew. Can you add them when you have a chance.

Ahh yeah, sorry. I've made a todo item.
BUG in MultiIndex.drop for not-lexsorted multi-indexes, #12078
State: closed

Closes #12078


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
I've taken into account the comments.

All green now.

merged via https://github.com/pydata/pandas/commit/f673af1bb51bb465144ffd2f843672e23433384a

thanks!
CI: latest ipython version for doc build
State: closed

See https://github.com/pydata/pandas/pull/12002#issuecomment-174550631

Doc builds are failing, testing if using the latest `ipython` version solves it.


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
lgtm. The build _looked_ like it succeded.

@jorisvandenbossche shall we merge this?

yes!

ok, merging now

dev docs built! looks good.

lots of warnings in the build of course....

and did API reference! awesome!

> lots of warnings in the build of course....

Yes, there is an issue with our custom accessor documenters that are not working anymore with the latest sphinx (that I needed for getting this run on travis). So need to investigate that ..

Apart from that, we normally don't have that many warnings! :-)
BUG: style.set_precision(0) displays spurious .0
State: closed

Fixes #12134


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
tests!

e.g. format a frame with a bunch of specific test data (e.g. like you have in the issue, but with NaN, infs, and any special cases).

Then assert that the contents of the generated HTML are what you expect

superseded by #12162
REF: reorganize pandas/tests/test_series.py
State: closed

Similar to test_frame.py, but only 8K lines to deal with this time. 


Labels:
name: Clean

Comments:
The sparse tests were relying on a small subset of unit tests in `CheckNameIntegration` (including several tests which didn't use non-sparse Series at all). I'll fix this up in the next day or two to get the build passing so this doesn't linger too long

@jreback Think I fixed the sparse tests, merge away after review when ready

thanks!
DEPR: GH10623 remove items from msgpack.encode for blocks
State: closed

closes #10623 


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
hmm legacy file for 0.17.1 not passing.  I think it's tz-related.  Will check later.

passing now.  had to make changes for `DatetimeTZBlock` (it wasn't tested?) and skip files packed in P2 with 0.17 :( due to #12142

updated  

fyi some of these fixes will prob address: https://github.com/pydata/pandas/issues/11455, maybe #11755 

updated... but I forgot to squash :(  will do that later

pls add a whatsnew as well (put in API changes).
make a note in `io.rst` that the writing format is changing in 0.18.0 so we won't have forward compat (IOW older version of pandas cannot read newer version)

this would close #12142 as well? (should add a warning in the docs about this) (e.g. show your matrix)

I can change all the keys to unicode, and then it will.  will update tonight

Ah half way through but have to go.  I will get it done tonight but is that too late? :(

sure no prob

updated.  manually tested with python 2.7 and 3.5 on linux (zlib and no compress). 

updated.  So this is how I tested P2 & P3 compatibility: use the script to generate legacy data for 0.18.0 in P2 and P3, put them in 0.18.0 folder, and let the tests pick them up.

thanks!

I added some more test msgpacks for 0.17.1 to cover the bases

https://travis-ci.org/pydata/pandas/jobs/109620001

I think we need to skip if blosc is not installed on these tests. 

can you do a PR for this?
PERF: add support for NaT in hashtable factorizers, improving Categorical construction with NaT
State: closed

closes #12077 

```
before     after       ratio
[1330b9fe] [404911c6]
17.25ms     1.21ms      0.07  categoricals.categorical_constructor_with_datetimes.time_datetimes_with_nat
```


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
Cool, kind of a no brainer. 

perfect example of why we need common NA handling in arrays. had to fix up factorize and rank! for int64

Yep. I'd be all for 100% bitmasks if it wouldn't cause legacy issues; maybe we'll get there someday
Fix Styler._translate failing on numeric columns
State: closed

Addresses https://github.com/pydata/pandas/issues/12125.

The previous behaviour was to use a `__getitem__` lookup which caused incorrect behaviour with numeric columns. The adjusted code uses `iloc` numerical indexing to resolve the issue.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
tests!

and u can use .iloc[r, c]

I'm working on the tests right now, just trying to figure out the best way to do "doesn't raise error" without comparing it to the output dictionary. 

Didn't know about the `.iloc[r, c]`, I'll change it to use that instead.

I've got a branch that changes the template and translate around a bit to be cleaner.  I'll push that later today when I'm at a computer and see how it messes up yourPR here. 

Let me know what happens and I'll rebase form your PR if this still exists. 

It might be worth keeping the test case anyway.

@HHammond I have got that change included on [this branch](https://github.com/TomAugspurger/pandas/tree/style-display-format). I've cherry-picked your test, thanks. I should have that branch wrapped up sometime this week.

Sounds good. Is it time to close this PR?

Sure, we'll leave the issue open though. Thanks.
CLN: reorganize index.py, test_index.py
State: closed

Split apart these two very large modules and created a new `pandas.indexes` subpackage. It would be nice to move all the index class code from `pandas.tseries` there in a followup patch.


Labels:
name: Clean

Comments:
Once the build passes, best not to let this patch linger (to avoid rebase conflicts). @jreback I don't think I broke anything but let me know if any concerns that I can look into

The python 2 tests are still failing, I think you missed a `from pandas.compat import range` import in the `range.py` file

Thanks, think I got it now. Will check the build in an hour or two here

@jorisvandenbossche @jreback merge when ready. thank you

OK, cleaned up things per your comments. 

lgtm. ping when green.

@jreback we are good to go

boo ya! thanks!
DEV: Add static modifier to inline declarations.
State: closed

It is currently impossible to build a debug version of Pandas on Mac OS, since some of the C function declarations are declared as `inline`, and compiling in debug mode turns off optimizations and hence inlining. While this could be solved by adding a compile argument to `setup.py` to use the C89 behavior for inline as in #10510, I believe an ultimately cleaner solution may consist of declaring inlined functions as `static`, so that the code is properly C99-compliant.

For the implementation, I added the `static` keyword to the definitions of `PANDAS_INLINE` and `P_INLINE` (and removed duplicate occurrences of `static` from some function declarations), so that future implementations that use those defines are automatically compliant. This is slightly different from e.g. Cython, which defines `CYTHON_INLINE` as `inline` and declares everything as `static CYTHON_INLINE`.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I think this provides an alternative solution for #10510, but I haven't tested this yet (currently downloading a FreeBSD vm over crummy internet).

Makes sense to me. Reviewed lgtm if the build passes. 

I ended up removing the conditional flag `-fgnu89-inline` on FreeBSD, since the addition of `static` makes it unnecessary. Building succeeds on my copy of FreeBSD 10.2, and the tests more or less run (there are some failures but they don't look related).

pls add a whatsnew note
and comment on that other issue for them to try this PR (just to confirm)

Done. Thanks guys for the quick feedback.

@jvkersch thank you sir!
BUG: Avoid cancellations in nanskew/nankurt.
State: closed

closes https://github.com/pydata/pandas/issues/11974

This replaces the implementation of `nanskew` and `nankurt` by an algorithm that does not produce round-off error due to subtracting very similar large values. The sample skew and kurtosis are computed directly from the moments, rather than expanding the moments, which leads to cancellation errors. The algorithm implemented here is about as efficient as the original implementation (in terms of time and memory); a more efficient one-pass algorithm is certainly possible but would require some Cython code.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
very minor change. pls rebase. ping when ready.

In computing skewness, is the computation of `m3` still potentially unstable?  (Just wanted to understand the new behavior)

@jreback Thanks for the quick review. I made the changes, and Travis passes.

@kawochen Not sure why you refer to `m3` explicitly, but while the calculation with the moments can also be unstable (e.g. when you have a series with huge values and a huge mean), it should be much less so than the previous implementation, which does things like `(x ** k).sum() - x.sum() ** k`, which quickly picks up a lot of round-off error.

@jvkersch I referred to `m3` because that was one of the calculations that could benefit from doing the positive and negative parts separately.  Symmetric samples would have a `0` skew more often.  But the final skew numbers of symmetric samples are close enough to `0` as they are, so I'm just pointing that out, but not suggetsing that we do that. 

@jvkersch thanks!

@kawochen I see, that makes sense; I didn't think about the alternate signs. Thanks for the clarification.
Allow from_tuples to accept and iterator
State: closed

Since many methods of constructing a list of tuples in Python3 return iterators, this flattens them. Although using from_arrays often works better.


Labels:

Comments:
tests and a usecase example pls.

That proposal also copies an existing list, which would have some performance impact where `tuples` were already a list.

You could have a function that checks if something in an iterator (or isn't a `Sized`), and coerces it to a list if an iterator. 
You could also try and delay checking the `len` until later, after the iterator would have been iterated, and see if users would get reasonable error messages where there is a break.

pls reopen if you wish to update according to comments

Thanks Jeff,

Will do if I have some time to complete.

Spencer

On 1/27/2016 11:00 AM, Jeff Reback wrote:

> pls reopen if you wish to update according to comments
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/pydata/pandas/pull/12119#issuecomment-175707418.
ENH: add error message for merge with Series GH12081
State: closed

closes #12081 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jreback incorporated your comments

@jreback true, looks much nicer now.

looks good. can you add a whatsnew note in API changes.

@jreback I added the note to whatsnew. I didn't manage to run Sphinx to test build the doc though. Can you give it a spin?

I'll have to clean my development environment and move from using pip to conda. Just don't really have the time now.

@dan-tee no looks good. thanks for the PR!
CLN: remove core/matrix.py, not imported into the pandas namespace
State: closed


Labels:
name: Clean

Comments:
+1

I don't think this needs to be deprecated in any way.

anyone disagree?

@wesm @jorisvandenbossche 

I'm +1 on removal. Deprecation warning ought to have been there in 2011 but at this point minimal users will be affected. 

you pushed the button!! =) 

oh right
hmm

is there a way to disable it in GitHub?

No; it's been requested many times by users, though, since in a lot of OSS projects with tighter merge protocol they've had mishaps with merges happening by accident
ERR: read_csv with dtype on empty data, #12048.
State: closed

ERR: read_csv with dtype specified on empty data 

closes #12048.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
pls add a whatsnew note in bug-fixes.

also, pls run the asv suite for parsing to see if any perf issues.

merged via https://github.com/pydata/pandas/commit/ab3291d7ddbe821733f199c6ce414d7a6b9c18e0

thanks!
Add Zip file functionality. Fixes #11413
State: closed

closes #11413

This PR leverages Python's ZipFile functionality to automatically unzip files read into DataFrames using read_csv().


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
as an aside, would be interested in solving issues brought up in https://github.com/pydata/pandas/issues/11666 and partially addressed in #11677 (not merged)

@jreback I would be happy to continue this to the other `read_*` functions. I like the idea of refactoring out the compression determination and the decompression steps, so that it can be used in all the `read_*` functions. I won't use the PR #11677 because it does not include tests or zip functionality. It also only focuses on `pickles`. 

@jreback Could you merge this in and I'll take care of the other requests?

@jreback could you please merge this before you make anymore changes to the tests. 

you need to rebase then I'll review

@jreback Rebased. Thanks.

you need to force push. you should only have your commits. see [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas)

@jreback I'm redoing this Pull Request. I will close this one and open a new PR.

ok, in the future, just push to the same one

Can the same functionality be added to read_fwf() and other read_\* methods?

#11666 is related
#12688 is where this could be done, e.g. the code is a spread out a bit. welcome for you to take a crack at it @itzybitzy 

IOW, the compression interfaces need to be pulled out a bit from the parser code

Sorry @jreback, I just don't have the skills to do that (yet). Is there something good for a beginner to work on? Documentation maybe? (Not even sure where the right place to ask this is.)

http://pandas.pydata.org/pandas-docs/stable/contributing.html

selects label of difficulty novice and you will see lots of issues
Update test_panel.py
State: closed

closes #12101

changed deprecated getargspec to signature for python 3 while keeping the getargspec for python 2


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Deprecate
description: Functionality to remove in pandas

Comments:
the compat changes need to be incorporated in this PR

merged via https://github.com/pydata/pandas/commit/0b8fa9dcdb6f47e40441467dd7ebb529914bb719

thanks!
PEP: pandas/core round 7 (window, reshape, series, format)
State: closed

And residual categorical.

What should we do with api/sparse/matrix as those are all unused imports--noqa for each line?


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
You can put flake8: noqa at the top of those files

lgtm. merging, thanks!
CLN: fix all flake8 warnings in pandas/io
State: closed

I suppressed warnings in test_parsers.py because of the amount of literal strings in the file. 


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@jreback can you give me a thumbs up to merge if the build passes? 

or @jorisvandenbossche too. Nested lists in RST I think require the extra line break

some minor comments. only issue is maybe we shouldn't touch.

`io/data.py`, `io/wb.py` & `io/ga.py` (and associated tests) and just put `no qa` at the top as these are deprecated.

I see; any harm in leaving them as is here or would you prefer I revert the changes? 

in theory your changes should be fine for `io/data.y` & `io/wb.py` (except for sure revert the `ga` as those tests have not been run for quite some time and are essentially disabled).

just trying not to touch those until they are removed.

@jreback I think we are not going to updat the ´io/data.py´ code anymore with changes that happen in pandas-datareader? (as we did in the beginning) If that is the case, I don't think there is any harm in leaving that file in here, otherwise I would say to revert (to more easily compare/update with pandas-datareader)

@jorisvandenbossche no it won't be updated. Just trying to not change it. These pep changes won't hurt,  as they are tested, so really no big deal. (bigger deal is `ga.py` which in theory should still work as hasn't been touched).

OK, I unstaged the ga/wb/data changes and cleaned up the parse_dates bit of the docstring. Will merge once the build passes
ENH: GH12042 Add parameter `drop_first` to get_dummies to get n-1 variables out of n levels.
State: closed

closes #12042 

Some times it's useful to only accept n-1 variables out of n categorical levels.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@jreback it might be tough it find a better name than `drop_first`, though agreed we usually go with `keep_` parameters...

We should define what happens when `drop_first` is True and there's a single level. Right now it fails with an IndexError I think (haven't tested). I'm not sure whether we should raise or not. Any idea what R does?

Looks like R raises

``` R
> year <- c(1, 1, 1, 1)
> year.f
[1] 1 2 3 4 5
Levels: 1 2 3 4 5
> year.f <- factor(year)
> model.matrix(~year.f)
Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]) :
  contrasts can be applied only to factors with 2 or more levels
```

although since pandas is more of a data manipulation library than a stats library, we might want to not raise an error. That said, I'm not sure what we'd return if we do catch the error, the single level or no levels at all?

I would raise a `ValueError` that doesn't really make any sense.

When `drop_first` is turned on and there is only one level, currently it will return an empty DataFrame:

``` Python
In [1]: import pandas as pd

In [2]: pd.get_dummies(pd.DataFrame(list('aaaaa')), drop_first=True)
Out[2]:
Empty DataFrame
Columns: []
Index: [0, 1, 2, 3, 4]
```

By doing this I think that the user doesn't have to check the levels contained in a column before using `drop_first`.
Otherwise if we raise an error it will be the caller's responsibility to check the levels of the columns they want to get dummy from. It may be painful to do this.

If you guys think raise an error is preferred, please comment and I will change the behavior 

@TomAugspurger any comments?

Just the minor doc typo. Looks good otherwise.

pls
squash when fixed 

@BranYang did you mean to close this?

@jreback I found that it's too messy when I try to squash my commits. So I just revert to a previous state and re-commit it.

that's fine. I will take care on merge anyhow.

@jreback pls kindly review the tests..

thanks @BranYang