BrandonFors/pandas-github-issues huggingface dataset:
Make pivot values work with most iterables
State: closed

Previously the values check for pivot tables was only looking for lists or tuples. This patch allows for any generators or other iterables (such as dict.keys() in python 3) to be passed as values. They will be converted to lists when they are identified as iterables, excluding strings. The performance hit from the extra list type assignment to iterables that are already lists or tuples should be minor.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
what issue is this supposed to address?

need tests

The problem is that right now the only objects that can be passed in as values to a pivot table are lists and tuples. I would like to expand this so that other iterable objects like generators and (in python 3) dictionary methods like `.keys()` can be passed in as well. My use case is something like this:

```
import pandas as pd
import numpy as np

df =pd.DataFrame({'A': ['foo', 'foo', 'foo', 'foo',
                             'bar', 'bar', 'bar', 'bar',
                             'foo', 'foo', 'foo'],
                       'B': np.random.randn(11),
                       'C': np.random.randn(11)})

print(df)

aggs = {'B':np.sum, 'C':np.mean}

pivot_output = pd.pivot_table(
    df,
    index=['A'],
    values=aggs.keys(),
    aggfunc=aggs,
)

print(pivot_output)
```

This works in Python 2, but returns an error in Python 3 because `.keys()` returns a `dict_keys` object instead of a list of keys. This forces you to explicity change it to a list before passing it through which feels clunky: `values=list(aggs.keys()),`

@MaximilianR pointed out that using `is_list_like` is probably preferable in the source, instead of the conditional I wrote. I am also wondering if its neccessary to expicitly convert other iterable objects to a list, or if they would work appropriately as is. I need to test more.

As far as unit tests go, I'm still learning and I'm not sure how the best way to write those for this would be, but I'll continue looking into it.

I may have jumped the gun on this PR a bit, so please let me know if theres a better way I should proceed.

In lieu of anyone more experienced answering your questions:
- If you can use one of the standard checks, that's great. `is_list_like` looks good.
- Unit tests are awesome. You can start with that use case if you want - just assert that the result is what you want. If you haven't run any nosetests before, have a look at the contributor docs. If you need help writing the tests, have a look at the existing tests, or post questions here.
- Coercing to a list _might_ be necessary, but probably isn't. Write the tests; if you have problems see if coercing helps. 

Congrats on being halfway to your first PR.

this needs tests

lgtm. pls add a whatsnew note in enhancements section, use this PR number as the issue number

thanks!

Thanks @jreback and @MaximilianR for all your help and suggestions on my first PR!
BUG giving modulo operator back to Int64Index
State: closed

Fix issue #9244


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jreback you were right, this also allows the use of the modulo operator for `Float64Index`

I've updated the tests and the what's new

merged via https://github.com/pydata/pandas/commit/4625d4d889842041b70f11ae16c0757590fef0d8

thanks!
TST asfreq / resample set freq correctly with regular timeseries
State: closed

Adding a test for issue #5613, which seems to have been solved


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype

Comments:
merged via https://github.com/pydata/pandas/commit/1d7521b32abf58c86193fdece042a64f7f6e5c01

thanks!
DOC/CI: include api docs on travis
State: closed

Continuation of #11069 (It seems I cannot reopen that PR because I already force pushed the branch ..)

Closes #3800

TODO:
- [x] remove printing of doc build log file in after_script


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Reason for reopening this, is that I noticed that sphinx 1.3 seemed to be improved in performance. And indeed, now the doc build on travis takes less than 20 minutes, while before it hit the maximum travis run time of 50 min.

@jreback Can you have a look at this? Note that there is still one temp commit that should be removed, so the full diff is not fully correct (and I also need to rebase)

This is related to https://github.com/pydata/pandas/issues/12003 (@shoyer), but I think this is still relevant. Maybe not necessarily for building the dev docs (if we start using RTD for this), but you can always check the doc build log for PRs this way.

```
./make.py
sh: 1: jupyter: not found
Running Sphinx v1.3.3
```

do you need to add jupyter to the install?

Rebased, removed the temporary commit, and added `nbconvert` as a dependency

FYI, I saw [nbsphinx](http://nbsphinx.readthedocs.org/en/0.2.3/) the other day. I ran it on our style notebook and it seemed to work pretty well. There were a few CSS things that don't get picked up properly, but we can probably move to that soon.

@jorisvandenbossche this looks good.

only thing (and can do later), is maybe move this build up so its the first one after main builds (before slow ones).

merge when ready (try out `merge-pr.py`)

@jorisvandenbossche 

build on master: https://travis-ci.org/pydata/pandas/jobs/104512895

failed apparently

Thanks for noticing. Probably due to the adding of nbconvert to requirements before I merged (what will maybe have given an incompatible nbconvert and ipython version, will need to investigate that further). 
I actually already failed in the PR, but because it's one of the allowed failures, I didn't notice ...

https://github.com/pydata/pandas/pull/12138
Small improvements to str_cat
State: closed

closes #11435 
closes #11334.

Added some doc examples to make the behavior with `nan` more clear, and catch an error condition when a `sep` is passed instead of an `others` (array) as an argument and print a more intuitive error message.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Strings
description: String extension data type and string data

Comments:
tests!

pls also add entries in whatsnew (one for each of the bug fixes). we'll call ignoring nans a bug fix.

can you rebase / update

I may have just botched this... tried rebasing on master and now there are many changes I didn't include reflected in the diff. 

http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas

`git rebase -i master`

then

`git push <yourremote> <yourbranch> -f`

You have to make sure your _your_ master is in sync with upstream master. 

You can revert to your original patch with "git reset --hard HASH" -- look in your git reflog to find the commit hash. 

This was botched and left to languish-- have restarted and will ref this in the re-do.
BUG: DataFrame.round() drops column index name #11986
State: closed

Fixes #11986. Used the same reconstruction method as `DataFrame.isin()` to ensure that column names are kept through rounding. Works with `np.round` as well.

Pretty minor fix -- does this need a release note or anything else?


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
tests!

+tests for `np.round` and `DataFrame.round` to ensure that the column names are preserved and that the rounding isn't otherwise affected.

Better like this?

yes although I wouldn't change it back to None (just change any expected after this)

can u update the series test as well (add a name)
it should work already

I'd put the new test in the middle of the rounding tests, so there would have been a lot of expected frames to fix without resetting the name. I've moved it to the bottom now, and added tests to ensure that naming the columns doesn't mess up `Series.round` or the name of a selected `Series` after a `DataFrame.round`.

pls add an entry in whatsnew for this bug fix.

gr8. ping when green.

Green. Thanks for the help with this, @jreback 

merged via https://github.com/pydata/pandas/commit/9b5c052b0807e4990dd5a8fe39238153c2ec9632

thanks!
BUG: Allow typecode of 'e' to be passed to select_dtypes
State: closed

Closes #11990.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
hmm why does his raise a TypeError? float16 is platform specific I think ?

np.e is defined and it's a float :)

```
In [1]: np.e
Out[1]: 2.718281828459045
```

add a whatsnew note as well

minor change, pls squash, ping when green.

can you update

Green.

I find your patch fine. But isn't this odd?

```
In [11]: df = DataFrame(np.random.randn(10,2))

In [12]: df[0] = df[0].astype('float16')

In [13]: df.dtypes
Out[13]: 
0    float16
1    float64
dtype: object

In [14]: df.select_dtypes(include=['float'])
Out[14]: 
          1
0 -0.458946
1  0.456845
2  1.246123
3  1.296991
4  1.445800
5  1.082351
6 -0.895652
7  0.031383
8  1.320909
9 -1.783126

In [15]: df.select_dtypes(include=['float16'])
Out[15]: 
          0
0 -0.501465
1  0.623047
2  0.207397
3 -0.251465
4  1.599609
5 -2.359375
6  0.354004
7 -0.191650
8 -1.067383
9 -0.903320
```

merged via https://github.com/pydata/pandas/commit/138b8ba853a6f4356b23d37d76e69425373f3315

That would be my expectation since `np.float64 == np.dtype(float)`, platform dependent. It's one of the reasons I ran into this, trying to use all float types to capture all eventualities. 

Actually, it might be nice to just accept the all typecode aliases too.

On Tue, Jan 12, 2016, 5:48 PM Jeff Reback notifications@github.com wrote:

> Closed #11991 https://github.com/pydata/pandas/pull/11991.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11991#event-513097549.

would accept PR for that addition
COMPAT: drop suppport for python 2.6, #7718
State: closed

closes #8760
closes #7718 


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
any complaints, comments, objections?

:smile:  `compat` will get 400 lines shorter

oh do I need to things out?

`OrderedDict` and `Counter`

This came up on the scikit-learn mailing list last week. I don't think they've come to a decision yet.

If we are approaching a 1.0 this year we should think about what we want to support indefinitely. I _think_ that's an argument for dropping 2.6.

Also, dict comprehensions :raised_hands:

yes, I need to make an issue for what we want to _add_ (also can use things like `.assertGreater` and such in testing).

@kawochen turned out to be 478 lines! (unless I took out too much)

puzzled while `ChainMap` is not in 2.7. It _says_ it has it, but not when I look. oh well.

```
In [115]: from ConfigParser import _Chainmap as ChainMap
```

wow, that's an odd place to have it!

If you search for 26, you will find a few more places that can be tweaked (e.g. windows build script)

@kawochen thanks.

updated

bombs away
Concat perf
State: closed

fixes #11958 


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
pls add a whatsnote in v0.18.0 on perf section.

squash as well

Done, please let me know if you need anything else.  Curious, any ETA yet on 0.18.0? (No rush)

thanks long when green

0.18.0 prob end of the month

All green.  Thanks @jreback 

merged via https://github.com/pydata/pandas/commit/3667b7636da878427c72dd59e1a5845e52ea306c

thanks!
BUG: Timestamp rounding wrong implementation fixed #11963
State: closed

closes #11963 


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
pls update `Timedelta` as well. need impl and tests.

looks good.

small comments. pls run `git diff master | flake8 --diff` and incorporate pep8 fixes.

ping on green.

I think it's okay

merged via https://github.com/pydata/pandas/commit/4437e9c9e161b884450e8027daa3622f74c4b9e7
ENH: add is_unique attr to Series
State: closed

Try to fix https://github.com/pydata/pandas/issues/11946


Labels:
name: Enhancement
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
add into api.rst as well

merged via https://github.com/pydata/pandas/commit/816a51f410965999c4c50c4b29d7355706e70a7f

thanks!
WIP/PEP8: pandas/core
State: closed

xref #11928 

Putting this out there as a start.  Will add to this over the next couple of days.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
just merged: https://github.com/pydata/pandas/pull/11941

will show the linting effect in travis (though not fail because of it).

FYI, I have found `autopep8` does a pretty good job (though doesn't fix long lines)

I am also ok with doing this in sections (3 files sounds about right). So when one is ready to go. pls ping.

green...there are still a few E731 lambda should be def, but substantially better than before.

@rockg can you specify what you exactly did? Is this the result of an automated tool (autopep8), or from manual clean-up?
Because I not fully 'happy' with all style changes, but if it is the result of a tool, then it will be probably easier to just accept the particular style of that tool (as style will always be something subjective).

Apart from that I put some comments of things that are not subjective.

Manual with the aid of editor defaults for line continuation, etc.  I'm happy to go through and change any persistent issues, but my main goal here was to result in no output from flake8.

As 90% of the issues are long-lines and autopep8 doesn't do that I felt it was easier to do manually.

`--max-line-length 79` as an option to autopep8

that reminds me - we should recommend autopep8

any way to setup default options? like we do for flake8

Looks like it

```
  --ignore-local-config
                        don't look for and apply local config files; if not
                        passed, defaults are updated with any config files in
                        the project's root directory
```

it seem to look for a file called `.pep8` https://github.com/hhatto/autopep8/blob/9eb1121f357077c7d71fc770e25d3678f906a401/autopep8.py#L122

Yes, seems that it does.  I was only going on the comment above.  Is it in such a state to use autopep8 for this set or only do that for future ones?

For eventual continuity with the rest of the source, I'll use autopep8 and repush.

can u add in the config file for autopep8 as well

thanks @rockg
Leading space in to_string(index=False), #11833
State: closed

closes #11833 

I had to create a new PR , due to unresolved conflicts
Just look at the third commit , it is all squashed , ignore the rest

I don't get it why the travis build is not being executed for the third commit, it fails for its parent(the first commit) , but would  pass for itself , as it is an updated version of its parent and the bug in the parent in tackled

I even checked for the merge-ability of the third commit , it merges with the branch. Still the travis build is not being executed

@jreback 


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
this still needs to be squashed. pls don't open another PR. you can simply do it in the _same_ PR. The instructions are [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas). Pls ask if you are having trouble.

merged via https://github.com/pydata/pandas/commit/bc0a166bdb30b828e1dac3aa1db57ea8fa50e644

thanks!
PEP8: add in flake8 checking
State: closed

don't actually fail this till #11928 is clean.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@jreback so this gets run before the actual build right? Is that where we want it? Or would we rather have the tests run, get the feedback about the tests, and potentially fail afterwards?

yeah could run it after

better? https://travis-ci.org/jreback/pandas/jobs/99880071

its at the end in its own section. It will pass for now (but show you the output)
head(0) and tail(0) return empty DataFrames
State: closed

closes #11930 

frame.head(0)  and frame.tail(0) now return empty DataFrames.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
make sure to test groupby! (need updating maybe)

Fixing the broken unit test....

groupby is already tested: https://github.com/pydata/pandas/blob/a89f48940cd0a7346302311011beb3a09e9be292/pandas/tests/test_groupby.py#L1912-L1913.

thanks! lgtm, but please squash.

I don't really understand why Travis is failing - it seems to be some kind of connection error?  Is that something I've done wrong?  

Looks unrelated. squash and maybe the tests will pass next time :)

I'm sorry, I'm not sure what you mean by 'squash'.

http://pandas.pydata.org/pandas-docs/stable/contributing.html#combining-commits

Ah, ok.  I think I did that right, hopefully the tests will pass this time. 

Need more thing, a release note in `doc/source/whatsnew/v0.18.0`, maybe under the API changes section, to document the change. You can squash that commit as well (which you did correctly). Thanks!

Added a note in the release docs.

ok, lgtm. ping when green.

Great!
EHN encoding parameter for to_latex
State: closed

closes #7061 

I'm working on solving issue #7061: currently, with Python 2 it is not possible to use `.to_latex()` when the dataframe contains utf-8 strings. This PR adds an `encoding` parameter to make it possible.

The work is not completely done though, and I'm making this PR to start a discussion here. In its current state, the PR quickly fixes the issue and provides tests, so it is kind of a minimal PR.

@jreback in the discussion on issue #7061, you made the following suggestion:

> you _may_ want to add a `LatexFormatter` (as a sub-class of `DataFrameFormatter`) as this will allow some re-factoring to be internally done later on. 

I would be glad to work in that direction, but I'm a bit confused by the current organization of the code. This `LatexFormatter` would seem to me to be similar to `HTMLFormatter`, which inherits from `TableFormatter` (exactly as `DataFrameFormatter`). In that case, `DataFrameFormatter` would use `LatexFormatter` for `.to_latex()` exactly as it uses `HTMLFormatter` for `.to_html()`.  Is that what you had in mind?


Labels:
name: Unicode
description: Unicode strings
name: IO LaTeX
description: to_latex

Comments:
@nbonnotte what I mean is that I think a `LatexFormatter` be created in `format.py` which is _like_ `HTMLFormatter` (its a sub-class of `TableFormatter`). So you don't have all of the latex formatting code living in `to_latex` itself (which instead just instantiates a class and calls `.write_result`, similarly to `to_html`).

This will allow easier refactoring later.

BTW, this does not solves the missing `decimal` parameter I mentioned in the discussion of issue #7061

@nbonnotte what was the `decimal` issue? (can't seem to find it in that issue)

I just meant the `decimal` parameter is still lacking, I didn't add it. I should do another PR to do that.

@nbonnotte create another issue for the `decimal` parameter (we are going to close #7061) with this PR

comments, ping when green.

I've created another issue for `decimal` (#12031)

Working on the docstring, I realized there was something odd: I was initializing `LatexFormatter` with a `DataFrameFormatter` and a variable `strcols` containing the result of `DataFrameFormatter._to_str_columns`.

I think it is simpler if the parameter `strcols` is removed and the call to `DataFrameFormatter._to_str_columns` is performed in `LatexFormatter`. You can see in the 2nd commit the changes (dc425cb0e959ccab2941cab27f329ec41cb768d7).

There is still one thing that bothers me: now I'm calling a "private" method (`_to_str_columns`) from outside (that is, from `LatexFormatter`). Is that a problem?

@nbonnotte no that's ok

I've squashed my commits then.

All green!

pls `git diff master | flake8 --diff`

@jreback should be good now ^^

merged via https://github.com/pydata/pandas/commit/3a832dfda9b56c16085679bccec230de67f71ce5

thanks!
Leading space in to_string, #11833
State: closed

closes #11833 

Final Commits


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@jreback  @jorisvandenbossche  it has finally passed the tests !!
Had to link with travis to run them locally
Please have a look at the commit

The issue was https://github.com/pydata/pandas/issues/11833

needs a note in whatsnew, with an example as this is techincally an API change.

@jreback   @jorisvandenbossche  do i have to update the what's new section in the documentation?
I have no experience with Pandas workflow, please guide me here?

http://pandas.pydata.org/pandas-docs/stable/contributing.html, specifically http://pandas.pydata.org/pandas-docs/stable/contributing.html#documenting-your-code for the whatsnew.

@TomAugspurger  @jreback 
I did the changes in the documentation
With the recent commit 

- some comments
- remove the html-styling file from the commits.
- squash into a single commit.

@jreback  is there anything else that needs attention , for the issue #11833 , I did the docs changes as directed by you 

need to remove the html-styling ipynb that is included
pls squash to a single commit

@jreback  I did the needfull ,ignore the last commit :)
It is not required 
@jreback please have a look

you need to squash: http://pandas.pydata.org/pandas-docs/stable/contributing.html#combining-commits
issue-11833
State: closed

Updated the format.py  file in pandas core directory

Function updated : to_string()

Class : DataFrameFormatter

.travis.yml also changed to for fewer builds


Labels:

Comments:
@jreback  @jorisvandenbossche  I again created pandas development environment from scratch, to send pull requests. Now i am able to do it comfortably, no longers mutiple PRs to be created
I am closing the previous PR , would work on this
Thanks for bearing with me, I am new to pandas code base

@Vutsuak16 don't remove the travis file, that defeats the purpose of the testing

instead you need to run the tests locally, and see what fails, and change that test (as that is what this PR is about)

@jreback I  did run the tests, locally , hence have modified the travis file,based on the test failures.

you need to fix test failures not remove tests

pls reopen when u have updated
fix for the issue-11833 #11903
State: closed


Labels:

Comments:
@Vutsuak16 Is there a reason you are making so many PRs? Please just update the original one https://github.com/pydata/pandas/pull/11903 by pushing to that branch

If you have any questions regarding the workflow, please ask any!

@jorisvandenbossche  the permission to push is being denied, from my command prompt , so i am not able to update the original PR

The error i am getting while pushing 
fatal: unable to access 'https://github.com/pydata/pandas.git/': The requested URL returned error: 403
Issue 11833-2
State: closed

updated the to_string() function of DataFrameFormatter(TableFormatter) class ,in format.py file, of the core directory.This is another commit . I hope this works
#11833


Labels:

Comments:
@jreback,  sir the issue was to remove the leading white spaces, which the commit does.
Yet the tests fail, because the leading white spaces are required for the assertion.

https://github.com/pydata/pandas/issues/11833

You already have an identical PR #11903, you can work further with that one.