BrandonFors/pandas-github-issues huggingface dataset:
[ENH] Add orient argument and split option to to_dict. (GH7840)
State: closed

closes #7840 

Unlike the implementation offered in #7840, this version preserves the option to pass prefixes to the `orient` parameter.

I also rewrote the docstring to use the more readable list format from `pd.DataFrame.to_json`.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@rpetchler ok!

pls add a release note in v0.15.0 (in the deprection section) and we can get this in.

pls squash: https://github.com/pydata/pandas/wiki/Using-Git

@jorisvandenbossche any other comments?

Hmm... squashed the commits but accidentally merged as well. Advice on how to undo all this?

Thanks for your patience--

`git rebase -i origin/master`  then delete all the uneeded commits, and f/s the others

For the rest looking good!

merged via https://github.com/pydata/pandas/commit/5995dc36a090c193bca20d18f7089503a8674bbb

@rpetchler I squashed and merged it. thanks!
BUG: sub-frame assignment of a multi-index frame breaks alignment
State: closed

closes https://github.com/pydata/pandas/issues/7655

```
>>> df
             jim                                  joe                                jolie                               
           first              last              first              last              first              last             
            left center right left center right  left center right left center right  left center right left center right
2000-01-03     0      3     6    9     12    15    18     21    24   27     30    33    36     39    42   45     48    51
2000-01-04     1      4     7   10     13    16    19     22    25   28     31    34    37     40    43   46     49    52
2000-01-05     2      5     8   11     14    17    20     23    26   29     32    35    38     41    44   47     50    53

>>> i, j = df.index.values.copy(), it[-1][:]
>>> np.random.shuffle(i)
>>> df['jim'] = df['jolie'].loc[i, ::-1]
>>> df.loc[:, ['jim', 'jolie']]
             jim                                jolie                               
           first              last              first              last             
            left center right left center right  left center right left center right
2000-01-03    36     39    42   45     48    51    36     39    42   45     48    51
2000-01-04    37     40    43   46     49    52    37     40    43   46     49    52
2000-01-05    38     41    44   47     50    53    38     41    44   47     50    53

>>> np.random.shuffle(j)
>>> df[('joe', 'first')] = df[('jolie', 'last')].loc[i, j]
>>> np.random.shuffle(j)
>>> df[('joe', 'last')] = df[('jolie', 'first')].loc[i, j]
>>> df.loc[:, ['joe', 'jolie']]
             joe                                jolie                               
           first              last              first              last             
            left center right left center right  left center right left center right
2000-01-03    45     48    51   36     39    42    36     39    42   45     48    51
2000-01-04    46     49    52   37     40    43    37     40    43   46     49    52
2000-01-05    47     50    53   38     41    44    38     41    44   47     50    53
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: MultiIndex

Comments:
thanks!
DOC: Some more vis cleanups
State: closed

Two things included:
1. Try to use ggplot style. If not, fall back to the previous 'default'.
2. Fixes some warnings in the doc build (part of https://github.com/pydata/pandas/issues/8234). There are some remaining warnings that I didn't have a chance to track down. Will see if I can later.


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
@TomAugspurger I am going to merge this, as I want to build the docs for the release candidate. The remaining warnings can then be fixed before final release.
Disallow Series indexing by DataFrame
State: closed

Fixes #8444.  I don't like importing DataFrame, but this was most obvious way to do it.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
gr8. ping on green

@jreback -- Green...30+ min for the slow tests.  I noticed the issue open for it and that would be a great fix.

thanks for the fix

yah I made it a bit faster but lots of tests just take a bit of time 

u r welcome to work on it!
CI: install version of miniconda corresponding to python version
State: closed


Labels:

Comments:
closing?

Why not use
`conda create -n test python=2.7`
`conda create -n test python=3.3`
`conda create -n test python=3.4`

as needed?  Same same miniconda can do either. I suppose you save a few MB of download.

closing pls reopen if/when updated
BUG: fix Index.reindex to preserve type when target is empty list/ndarray
State: closed

This PR fixes #7774 and also includes:

TST: check index/columns types when doing empty loc/ix tests

CLN: don't _ensure_index in NDFrame._reindex_axes, it is done in Index.reindex


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Whoops, this PR is based on unmerged #8460 branch and shows two commits.

pls close the incorrect one, and rebase

ok rebase

 thanks.

the corner case with `s.reindex(iterator)` that made me add `_ensure_has_len(seq)` should have minimal impact, but I need to benchmark that. 

looks good
ready to merge?

FYI: https://github.com/pydata/pandas/commit/bc6a1f79963bdee03b0fd770a80b013c56743da3

the `np.int_` types don't compare correctly on windows (as they are 32-bit by default)

Oh, I never get win32 right :)

haha, its a beast!
BUG: fix Index.reindex to preserve name when target is list/ndarray
State: closed

This PR fixes #6552 and also includes:

CLN: drop copy_if_needed kwarg of Index.reindex, it's True everywhere
TST: enable back empty-list loc/ix tests that failed before
DOC: Bump Index.reindex docstring


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
looks good, ping on green

green. do you want me to add that GHXXXX tag on the test now?

yep

just rebase and add that tag
after u push just ping and can merge

rebased & added

thanks!
BUG: type diversity breaks alignment
State: closed

closes https://github.com/pydata/pandas/issues/8258, but more generally, on pandas master, assignment with indexer will align _both rows and columns_ if all columns have the same type:

```
>>> cols, idx = ['jim', 'joe', 'jolie'], ['first', 'last']
>>> vals = np.arange(1, 7).reshape(2, 3, order='F')
>>> df = DataFrame(vals, columns=cols, index=idx)
>>> left, rhs = df.copy(), - df.iloc[::-1, -2::-1]
>>> print(left, rhs, sep='\n')
       jim  joe  jolie
first    1    3      5
last     2    4      6
       joe  jim
last    -4   -2
first   -3   -1
>>> 
>>> left.loc[:, ['jim', 'joe']] = rhs
>>> left
       jim  joe  jolie
first   -1   -3      5
last    -2   -4      6
```

However, if I change the type of a column even _outside_ the section which gets assigned to, _only columns_ are aligned:

```
>>> df = DataFrame(vals, columns=cols, index=idx)
>>> df['jolie'] = df['jolie'].astype('float64')
>>> left, rhs = df.copy(), - df.iloc[::-1, -2::-1]
>>> print(left, rhs, sep='\n')
       jim  joe  jolie
first    1    3      5
last     2    4      6
       joe  jim
last    -4   -2
first   -3   -1
>>> 
>>> left.loc[:, ['jim', 'joe']] = rhs
>>> left
       jim  joe  jolie
first   -2   -4      5
last    -1   -3      6
```

Code-wise this is a bug, because [this line](https://github.com/pydata/pandas/blob/fccd7feabfe0bf15c30d712f3a4a4ff71e76ad4a/pandas/core/indexing.py#L446) is taking the _entire frame_'s index and re-indexes to that, regardless of the actual indexer which selects a section of the frame to assign to. That said, there are two places in tests where they rely on this behaviour, [here](https://github.com/pydata/pandas/blob/fccd7feabfe0bf15c30d712f3a4a4ff71e76ad4a/pandas/tests/test_frame.py#L1405) and [here](https://github.com/pydata/pandas/blob/fccd7feabfe0bf15c30d712f3a4a4ff71e76ad4a/pandas/tests/test_indexing.py#L1033).


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
when fixed / repushed and green, pls ping

@jreback done!

thanks!
PERF: optimize storage type for codes in Categoricals (GH8453)
State: closed

closes #8453

So easy enough to optimize the dtype of the codes array depending on the number of categories. Most
of the time will simply be `int8`. So pretty good savings.

This doesn't completely solve the storage issues as currently `factorize` returns a `int64`, but can deal with that later (as might also need some more cython definitions).

And of course in the extreme case (where len(categories) == len(codes) you are better off with an object dtype).

```
In [1]: s = Series(['foo','bar']*1000)

In [2]: s.nbytes
Out[2]: 16000

In [3]: s.astype('category').nbytes
Out[3]: 2016

In [5]: s.astype('category').values.codes
Out[5]: array([1, 0, 1, ..., 0, 1, 0], dtype=int8)
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
cc @JanSchulz
cc @shoyer

Maybe add a test with a case where you need a change in dtype when adding categories?

@jorisvandenbossche @shoyer all suggestions done

LGTM :+1: 

have to fix some stuff later
setitem breaking on invalid coercion

@JanSchulz suggestions taken. pls have a look (added some docs too)
Remove DataFrame.delevel
State: closed

Part of #6581.  Remove DataFrame.delevel which was previously deprecated in version 0.7.
@jreback 


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
This displayed issue wrong: http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#deprecations

any idea where the original deprecation came from?

if you can find it great, if not let's just remove the reference (to #420)

thanks

I am going to take it out (so if you find the correct reference pls do a PR to add back)
Add support  of 'decimal' option to Series.to_csv and Dataframe.to_csv
State: closed

closes #781

The 'decimal' option exists for read_csv method but not yet in 'to_csv' methods.
The lack of this option is particulary painful when we _have to_ work with Excel with European regional settings. 

This modification add this option to both Series.to_csv and Dataframe.to_csv.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO CSV
description: read_csv, to_csv

Comments:
along those lines, would be nice to add a vbench for this as well (and then you'll know!)
see [here](https://github.com/pydata/pandas/wiki/Performance-Testing)

@jreback Can this be revisited? 

Is there something that needs to be changed by @bertrandhaut (apart from adding a vbench)

I think this needs a test?

As suggested, I've added a test for this new option.

Joris' comments taken into account

@jreback Is this good to go? (not really familiar with this)

@bertrandhaut Before merging, we ask to squash the commits into one. Can you do that (https://github.com/pydata/pandas/wiki/Using-Git#fetch-and-then-rebase-interactively-to-squash-reword-fix-otherwise-change-some-commits, if you have questions, just ask!)

@bertrandhaut can you do a vbench run on the csv's to make sure that perf is unchanged, see here:https://github.com/pydata/pandas/wiki/Performance-Testing

For the vbench I tried but without success. For information I've only python3 on my computer and it seems that vbench is still not ported to python3.

About the vbench, I think that is correct that is not yet ported to python 3.

@bertrandhaut can you rebase and squash to a single commit. looks ok otherwise to me.

@bertrandhaut You were indeed working on master. In the future, better to always first make a branch if you are going to work on a certain feature/fix. But for this PR, it is OK.

Given you are working on master, this should work to rebase/squash:

```
git fetch upstream
git rebase -i upstream/master
```

merged via https://github.com/pydata/pandas/commit/671c4b3c52a02ee3736752b2645d08762c0fedaf

thanks!
API: SQL class definitions renaming
State: closed

WIP, related to #7960
- [x] renaming of 
  - PandasSQLAlchemy -> SQLDatabase
  - PandasSQLTable -> SQLTable
  - PandasSQLLegacy -> SQLiteDatabase
  - PandasSQLLegacyTable -> SQLiteTable
  - SQLDatabase.read_sql -> read_query
- [ ] adding docstrings to public functions of SQLDatabase
  - [x] read_query, read_table, to_sql
  - [ ] has_table, drop_table, ..
- [ ] add docs on use of SQLDatabase
- [ ] add api docs for SQLDatabase
- [x] rename some mentions of 'legacy' to 'fallback'


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jorisvandenbossche we sched to remove the old sql support in what 0.16?

Should I add deprecations for the old names? (they were never really public, so for now I just removed the old names)

I am going to leave the docs part of this PR for later (can do after rc release) 
For the removal of old sql support, I don't think we explicitely scheduled that, but I would say 0.16 or 0.17.

ok on old sql support (add to the deprecations list if not already on it)

If these were not public then I don't think a big deal (you can add a doc note if you want)
DOC: create text.rst with string methods (GH8416)
State: closed

closes #8416 


Labels:
name: Docs

Comments:
@jorisvandenbossche @hayd ?

This looks like a great change - it's also reducing the rather unwieldy size of the "basics" section. So, +2.

@davclark the question is what else, if anything should go in here? 

To my knowledge, you've moved the _only_ narrative documentation about string types. So, there's nothing else that should be moved. A common point of confusion for beginners is the auto-regexp functionality in many of these methods. So a highly visible message about that point could be useful both in the "getting started" and at the top of this new section. But I think this should be a separate pull request if you want me to draft it, right?

sure u can update this one
or push on top

@jreback I sent you a pr to your str branch.

@jorisvandenbossche ?

Yep, also +1 on seperating this into a seperate file.

Added two minor comments. And I was also thinking, what I am missing at the moment is a subsection about the indexing capacity of `str`:

```
`In [1]: s = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat']
)

In [2]: s
Out[2]:
0       A
1       B
2       C
3    Aaba
4    Baca
5     NaN
6    CABA
7     dog
8     cat
dtype: object

In [3]: s.str[0]
Out[3]:
0      A
1      B
2      C
3      A
4      B
5    NaN
6      C
7      d
8      c
dtype: object

In [4]: s.str[1]
Out[4]:
0    NaN
1    NaN
2    NaN
3      a
4      a
5    NaN
6      A
7      o
8      a
dtype: object
```

@jorisvandenbossche ok, let me put this up and can add to it (I'll add something like your example as well).
API/BUG: a UTC object inserted into a Series/DataFrame will preserve the UTC and be object dtype (GH8411)
State: closed

closes #8411 


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
cc @rockg
cc @ischwabacher
cc @adamgreenhall

This actually wasn't very difficult and is not really a very big change IMHO (as rarely would you actually do this). Came up because couldn't chain ops like this:

`s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')`
Categorical doc fixups
State: closed

Minor doc updates as part of #8375
- docstring in in cut/qcut + docstring examples #8077 (comment)
- whatsnew: In the note change "levels" to "categories" and s/Schultz/Schulz/


Labels:
name: Categorical
description: Categorical Data Type

Comments:
@jorisvandenbossche can you have a look at the docstring fixes?

yes, looked at them and looking good!

thanks!
BENCH: programmatically create benchmarks for large ngroups (GH6787)
State: closed

closes #6787

Uses ngroups=10000 as suggested in the issue, which takes about 1 hour on my desktop.

For results (vb_suite.log, pkl file) see: https://gist.github.com/dlovell/ea3400273314e7612f6e
Note: gist references a different commit hash.  I changed the commit message and added modification to doc/source/v0.15.0.txt, but the actual modifications to vb_suite/groupby.py are principally the same.


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
@dlovell  this is gr8!

can u add small but for small groups as well? (say 100?
and name very similarly)

might then have some duplicates (well some that do very similar - so can eliminate those and just use your new ones)

pls also create an issue to investigate the perf of some of these (just paste you test run in  new issue for reference) 

Commit c494c03 runs the tests for ngroups values 100 and 10000.  Output looks like

```
groupby_ngroups_100_describe                 | 780.9353 |
groupby_ngroups_10_cumcount                  |   0.3180 |
```

@dlovell thanks!
ENH: First attempt at adding coveralls
State: closed

First attempt at adding coveralls to pandas


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
See also this issue: https://github.com/pydata/pandas/issues/6251#issuecomment-42772760 and the linked code

Oh, well looks like you've already got coveralls in place... Are you planning on PRing that code?  I would also suggest adding `--cover-package=pandas` to your coverage branch
BUG: Panel.fillna with method='ffill' ignores the axis parameter (GH8251)
State: closed

Adds support for method-based fillna along the specified axis of a Panel. Also adds some additional validation when calling fillna on DataFrame.


Labels:

Comments:
hah not sure we has this kind of collision before!

@staple thanks!

welcome to work on other issues of course!

@jreback, thanks. I'll take another look at the "Good as first PR" tickets.

Hah... sorry about that. I'm incorporating the new tests you wrote for sure. Thanks!
add support for numpy 1.8+ data types for conversion to r dataframe
State: closed

Numpy changed the names of their data types in 1.8
http://docs.scipy.org/doc/numpy-dev/user/basics.types.html

specifically, bool, int, float, and complex are now bool_, int_, float_ and complex_

this requires changing the VECTOR_TYPES and NA_TYPES dictionaries used by convert_to_r_dataframe in pandas/rpy/common.py

numpy 1.7 isn't feasible to include as a project dependency (too many packages require 1.8)

Unfortunately, bool_ etc don't exist in numpy < 1.8, so you'll have to try/except that bit. I can change it and submit a pull request if you like


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
minor comment other wise looks fine
pls add a release note in v0.15.0 API section (reference this pr)

looks good. pls squash into a single commit and good to go.

squashed

gr8 ping me when green

green

@benjschiller thanks!
BUG: maintain order in excluding categories
State: closed

should fix current travis fails on master branch.

i cannot add tests right now because `test_remove_categories` in `test_categorical.py` does not fail on my machine. it also passes on a number of travis builds, and fails only on a couple.


Labels:

Comments:
[all green](https://travis-ci.org/pydata/pandas/builds/36359682) except for one IO error (`URLError`) unrelated to this patch.

well I already have a patch for this with some other code.

thanks anyhow

> thanks anyhow

@jreback thanks anyhow to you,

even if you have a patch for this shouldn't you respect the time spent for the PR?

I mean if i had seen that someone has already submitted a PR for this, that was a different story. but if i spend couple of hours on this but then someone just come and close it because "i have my own patch for this" that is just annoying!

@behzadnouri well I did this a couple of days ago, but was out for a jewish holiday yesterday. so getting around to patching it now.

thanks for the effort. I merged the cats stuff knowing their was a problem bug didn't have time to fix.

but as I have said to you many times before.
PR's w/o tests are useless. and waste MY time. in the future pls respect that.

@jreback 

-1- pandas master [started failing](https://travis-ci.org/pydata/pandas/builds) exactly 16 hours ago, and this patch was targeting that _very exact_ fail (you know _one patch_ for _one fix_ thing). so not sure how exactly you worked it out couple of days ago

-2- in this case, there were tests already failing on travis build. so technically the tests were already there. you do not agree?!

as i told you this is just annoying and disrespectful that you do not care for the time/effort spent for a pr and close it with a "my own patch" argument. you may want to work on that.

@behzadnouri not going to argue with you any more. Pls submit fully functional patches with tests / release notes / docs.

you contributions are appreciated, but I have noticed that you seem to be arguing (with not a lot of constructive criticism). Pls check that.

@jreback 

for the record: [this](https://github.com/pydata/pandas/pull/8396/files) is the patch i had submitted, and [this](https://github.com/pydata/pandas/commit/df3224ffa05791179d6e3c546e1f917a9a2a81f0) is yours;

-1- contrary to what you said above:

> I already have a patch for this with some other code.

there is absolutely no other code.

-2- there are absolutely no new tests

-3- I also had extra code comments in my patch which you have removed them.

this is just wrong on many levels.

@behzadnouri you are beating a dead horse.

I had this patch queued up since I had merged the original categoricals. It was supposed to be included but wasn't amended property.

get over it
BUG: apply mask to aligned new values
State: closed

closes https://github.com/pydata/pandas/issues/8387


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
since their are no tests how can u even tell if it works

@jreback i have tested locally. i add tests once travis comes green. currently travis fails on master. 

test driven development means tests first 
then a fix

pls do it like this once pulled

you can of course so it how you would like before that - when you pull it should at the very least have tests 

@jreback test added

merged via https://github.com/pydata/pandas/commit/1aa7c2c94170f5a0fd0daf3c90cb89f1aa85d8c3

thanks
DOC: Use square figsize for pie plots
State: closed

First commit closes https://github.com/pydata/pandas/issues/8308

second closes https://github.com/pydata/pandas/issues/8312 aside from using mpl 1.4 to build the docs.

I can post any examples if you want.


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
@TomAugspurger ?

Squashed down to one commit. @jorisvandenbossche these should take care of everything for the doc build other than using mpl 1.4 to fix the gridlines.
non-existent columns in DataFrame.dropna subset argument now raises KeyError
State: closed

closes #8303. 

 Now instead of interpreting a bad column name as the 'last column' of the `DataFrame`, `dropna` will raise a `KeyError` instead.  unit test and release note now included.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@mcjcode thanks!