BrandonFors/pandas-github-issues huggingface dataset:
Tests for ABC Types
State: closed

closes #10828

I used self.assertIsInstance(obj, type) and inlined as many of the expressions as I could while staying PEP8 compliant. I added in the two tests that I took out, that was an accident. Let me know if this looks okay!


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@DanNi0130 normally you don't need to open a new pull request, just update the existing one.

see contribution docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

small comment, pls squash. ping when green.

You want me to add the issue number as a comment in the code? I just fixed the function name.

yes

Done!

as i said. pls squash to a single commit. ping when travis reports green.

merged via https://github.com/pydata/pandas/commit/18012f0d42bb46f618e2cab8ab04620a1fe92d00

thanks!
removes categories & ordered from CategoricalIndex attributes
State: closed

`categories` and `ordered` are properties not attributes ([index.py#L3085-L3091](https://github.com/pydata/pandas/blob/3e35c842f37265ac7339908869d4cc7bdef784d9/pandas/core/index.py#L3085-L3091)). The side effect is that the categories are rehashed too often:

on master:

``` ipython
In [1]: from pandas.core.index import CategoricalIndex

In [2]: np.random.seed(2718281)

In [3]: n = 100000

In [4]: val = map('{:->4}'.format, np.random.randint(0, 10000, n))

In [5]: idx = CategoricalIndex(list(val))

In [6]: %timeit idx[::-1]
100 loops, best of 3: 14.1 ms per loop

In [7]: %timeit idx[idx.argsort()]
10 loops, best of 3: 22 ms per loop

In [8]: cProfile.run("idx[::-1]", sort=1)
         174 function calls (171 primitive calls) in 0.025 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.014    0.014    0.014    0.014 {method 'lookup' of 'pandas.hashtable.PyObjectHashTable' objects}
        1    0.004    0.004    0.004    0.004 {pandas.algos.take_1d_object_object}
        1    0.003    0.003    0.003    0.003 {method 'map_locations' of 'pandas.hashtable.PyObjectHashTable' objects}
```

on branch:

``` ipython
In [6]: %timeit idx[::-1]
The slowest run took 6.78 times longer than the fastest. This could mean that an intermediate result is being cached
10000 loops, best of 3: 29 µs per loop

In [7]: %timeit idx[idx.argsort()]
100 loops, best of 3: 9.38 ms per loop
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
yes that was some left-over code. thxs.
[DOCS][10MINS] Fix method for the 'where' operation example
State: closed

Fix method of the 'where' operation for getting.


Labels:
name: Docs

Comments:
@moutai This is in the 'Boolean indexing' section, so I think it is on purpose that it is not using `where`.

But you could e.g. add something like "which is equivalent to `df.where(df > 0)`"

I see, that makes sense in the Boolean indexing section. Please ignore.
BUG: Error while saving DataFrame with TimedeltaIndex to .csv #10833
State: closed

Fix in accordance with https://github.com/pydata/pandas/issues/10833


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv
name: Timedelta
description: Timedelta data type

Comments:
needs some tests. use the example you gave. add into `test_frame.py`. needs a round-trip tests. Note that you will have to read it as `object` dtype and the run it thru `to_timedelta` to compare against the original.

pls also add a whatsnew note.

It would be nice to refactor `test_frame.py` code... Btw, there are two existing assert methods: `self.assertXXX` and `assert_frame_equal`. I've used the first one as it seems like the second is to be reworked.

@soupault yes of course `test_frame` could use a refactor. but that's another PR!

thank you sir!
DataFrame to dict with index orientation.
State: closed

The method `DataFrame.to_dict` didn't have an option for index orientation unlike its cousin `to_json`, which does have.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design

Comments:
hmm, seems resonable. can you add a note in whatsnew / enhancements. (also add a versionadded tag in the doc-string).

Just added the note and the versionadded tag you asked. Should I've opened an issue before submit the pull request?
I was using pandas in a project and noticed the enhancement that method could have. Then I just thought in contribute, but I was unaware of the conventions.

minor corrections. pls squash. ping when green.

merged via https://github.com/pydata/pandas/commit/73dcb953efd1646fc1c53540637dde058f36fe92

thanks!
Fix handling of EOF in 'c' csv parser
State: closed

Fixes GH #10728
fixes #10548.

Also fixes:
- '\r' followed by EOF should be considered a blank line
- Escape character followed by EOF should produce an exception
- Line containing only whitespace should be skipped if `skip_blank_lines` and `delim_whitespace` are both `True`


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
The test on python2.6 fails because of a bug in the csv module that got fixed in 2.7 (see http://bugs.python.org/issue16013). What's the right thing to do in this case? Check the python version and skip the test if < 2.7? Or just get rid of this test altogether?

you can just skip in 2.6
I think there is a function in utils.testing for this (or if not create a _skip for 2.6)

lgtm, tiny doc change. ping when green.

lgtm. ping on green.

Tests are green

thank you!
BUG: fix bounds for negative ints when using iloc (GH 10779)
State: closed

This fixes the issues raised in [GH 10779](https://github.com/pydata/pandas/issues/10779).
closes #10547 

Previously, negative integers used in `iloc` were not stopped from going beyond the bounds of the Series. For instance, if `s = pd.Series([1,2,3])` then `s.iloc[-4]` returned bytes from memory outside `s`. This will now raise an IndexError instead.

Additionally, it was not possible to access the first row (index 0) of a Series or DataFrame with a negative integer in a list. For example, `s.iloc[[-3]]` raised an IndexError instead of returning a Series containing the first row of `s`. The behaviour of `iloc` has been changed to allow this.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
I think this also closes #10547. Can you add that as an additional test (and add to the release note)

@jreback I've added the additional test and information and travis has passed. Let me know if there's any more work to do...

thank you @ajcr !
BUG: Allow read_sql_table to read from views
State: closed

Solves #10750 

Follow-up from #10771

'read_sql_table'  will now allow reading from views. 

Added also a note in whatsnew and a unit test to check the behaviour.


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks!
BUG: pd.Series.interpolate(method='spline') Errort Msg, #10633
State: closed

Closes #10633

This a first attempt at the error messages mentioned. Not ready to merge yet but I wanted to start the conversation... 
I think I might have the error message wrong - anyone got any ideas. 


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Look like I need to add a test for k - or something or am i wrong?

I think that's ready to be merged :) 

needs a test

Done :) 

pls add a whatsnew note in bug fixes

I added a work through etc - maybe it needs an error message (another PR) for order=0 or order>=6

Updating on the basis of your feedback. 

I assume this one will be ok for the 0.17 release right?

ok, pls rebase and squash. ping on green.

Looks like this one got messed up a bit - do you have any suggestions on how to fix this?
I've tried rebasing to upstream - which is the pydata/pandas branch. 
I made the mistake here of not branching correctly from the start. 

Actually this seems to not be running through travis. I think I pushed to the wrong location. I'll fix it later.
Todo: Push via travis correctly

If you do 

```
git fetch upstream
git rebase -i upstream/master
git push -f origin/master
```

everything should be fine

Upstream should be my form right? Or the pandas original one?
ᐧ

On Fri, Aug 21, 2015 at 1:17 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> If you do
> 
> git fetch upstream
> git rebase -i upstream/master
> git push -f origin/master
> 
> everything should be fine
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10800#issuecomment-133374965.

## 

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

upstream should be 'pydata/pandas', origin is 'springcoil/pandas'

You can check this with `git remote -v`

Cool thanks for that reminder.
I'll check that after work :)
ᐧ

On Fri, Aug 21, 2015 at 1:23 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> upstream should be 'pydata/pandas', origin is 'springcoil/pandas'
> 
> You can check this with git remote -v
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10800#issuecomment-133376434.

## 

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

Thanks @jorisvandenbossche 

replaced by #10880 

normally don't open a new PR as you can simply replace all of the commits. but ok for now.

Accidentally done

On Friday, 21 August 2015, Jeff Reback notifications@github.com wrote:

> Closed #10800 https://github.com/pydata/pandas/pull/10800.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10800#event-389161182.

## 

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com
Allows set_index() to take an Index as an argument (#10797)
State: closed

Simple change, closes #10797 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design
name: MultiIndex

Comments:
this needs some tests. 

@michaelbilow can you add a test / whatsnew note 

closing, but if you'd like to update, pls reopen
TST: Suppress warnings of drop_duplicates tests
State: closed

Follow-up of #10236. Suppress unnecessary warnings during the test.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I don't think this is the correct fix. You shouldn't need to specify the `warnings.simplefilter('always', FutureWarning)` everywhere. This is done by `assert_produces_warning`. Or am I missing something?

Correct, fixed to use `clear` argument of `assert_produces_warning`.

@sinhrks do you know why the `clear` is needed in this case (otherwise we almost never use it)? 

lgtm, merge when ready.

Actually `clear` is not needed if we use `assert_produces_warning` in all cases. 

@jorisvandenbossche Pls merge if looks OK. 

ah yes! Merging, thanks!
BUG: Index.take may add unnecessary freq attribute
State: closed

`Index.freq` should raise `AttributeError` in non-datetime-like index. But `Index.take` adds `freq=None` and broke the behavior. 

```
import pandas as pd
idx = pd.Index([1, 2, 3])

idx.freq
# AttributeError: 'Int64Index' object has no attribute 'freq'

idx.take([True, False, True]).freq
# None
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jreback Thanks for the great advice. Fixed to use `.values.take` and `.asi8.take` in datetime-like case.

Mainly `iloc` gets faster.  I've also added benches for `take` (`series_take_intindex` and `series_take_dtindex`), but results look not significant in these cases. `series_xs_mi_ix` and `frame_xs_mi_ix` results look random.

If it looks reasonable, I'll add release not for perf improvement also.

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
series_iloc_list_like                        |   0.1010 |   0.8353 |   0.1209 |
series_xs_mi_ix                              |   1.2943 |   2.4173 |   0.5354 |
frame_xs_mi_ix                               |   1.3490 |   2.4823 |   0.5434 |
series_iloc_array                            |   3.4820 |   4.0780 |   0.8539 |
frame_reindex_upcast                         |   5.9937 |   6.9266 |   0.8653 |
frame_ctor_list_of_dict                      |  59.9620 |  67.9890 |   0.8819 |
frame_dropna_axis1_all                       |  68.5070 |  75.5123 |   0.9072 |
frame_dropna_axis0_any                       |  22.2783 |  23.9987 |   0.9283 |
series_take_intindex                         |   2.7970 |   3.0077 |   0.9300 |
strings_startswith                           |   3.6437 |   3.8893 |   0.9368 |
strings_endswith                             |   3.6343 |   3.8550 |   0.9428 |
series_take_dtindex                          |   2.7507 |   2.9174 |   0.9429 |
...
timeseries_timestamp_tzinfo_cons             |   0.0106 |   0.0090 |   1.1858 |
timeseries_infer_freq                        |  11.5213 |   9.6280 |   1.1966 |
timeseries_is_month_start                    |   3.1753 |   2.6217 |   1.2112 |
timeseries_custom_bday_cal_incr_n            |   0.0410 |   0.0330 |   1.2434 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

@sinhrks sounds good.

Updated the release note. Pls review.

lgtm
BUG: `read_sql_table` can't find views
State: closed

closes #10750 

Add ability to 'read_sql_table' to read views and implement an unit test to check its behaviour


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Looks good!
- Can you add a small note in the whatsnew file (in the 'other enhancements' section)
- You included the test in `_TestSQLApi` which only tests the API with sqlite. I think this is OK (I don't see potential differences between database flavours), but in that case the sql code for the different flavors is not needed.

@IamGianluca for future reference, it is not needed to open a new PR if something went wrong. You can always rebase the current branch and force push again.
BUG: Fix dtypes order when ordering is different from original file in pandas.io.stata.read_stata
State: closed

This fixes http://stackoverflow.com/questions/31783392/pandas-adding-columns-to-filter-will-mess-with-data-structure
The dtypes of the DataFrame returned by read_stata have the same order as the original stata file, even if columns are specified with a different order.


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
@jreback much cleaner now

lgtm. minor doc comment. ping when green.

@jreback good to go

thank you sir!
Add kwargs for kde in scatter_matrix
State: closed

Minor change to allow customizing the kde through the pandas api.


Labels:
name: Visualization
description: plotting

Comments:
Have you seen [seaborn](http://stanford.edu/~mwaskom/software/seaborn/tutorial/axis_grids.html#grid-tutorial)? It handles this type of plotting much better than pandas. Specifically, this plot is covered by [`pairgrid` / `pairplot`](http://stanford.edu/~mwaskom/software/seaborn/tutorial/axis_grids.html#plotting-pairwise-relationships-with-pairgrid-and-pairplot)

@TomAugspurger good point. Yes, I use have used it but I often forget to use it.

@TomAugspurger status of this?

@TomAugspurger close?
TST: test_packers.TestMsgpack checks for minimum structure and extra …
State: closed

…keys

closes #10732 

I wasn't able to reproduce the odd behavior.  Let's see if this passes on Travis. 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
yeh this is not a problem, however I mistakenly did this:

IOW this is a non-existing path and you simply get the btyes of the string back. (which is the defined behavior), though a bit unexpected.

```
In [9]: pd.read_msgpack("pandas/io/tests/data/legacy_msgpack/0.16.2/0.16.2_x86_64_linux_3.4.3")
Out[9]: 
[112,
 97,
 110,
 100,
 97,
 115,
 47,
 105,
 111,
 47,
 116,
 101,
 115,
 116,
 115,
 47,
 100,
 97,
 116,
 97,
 47,
 108,
 101,
 103,
 97,
 99,
 121,
 95,
 109,
```
BUG: Fix dtypes order when ordering is different from original file in pandas.io.stata.read_stata
State: closed

This fixes http://stackoverflow.com/questions/31783392/pandas-adding-columns-to-filter-will-mess-with-data-structure
The dtypes of the DataFrame returned by `read_stata` have the same order as the original stata file, even if `columns` are specified with a different order. 


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
looks good.
- minor code change
- pls add a note in whatsnew (use this PR number)
- squash

Done, I got rid of the `matched` dict as well because it doesn't do anything.

lgtm

pls rebase and squash (FYI move the release note higher up in the bug fix list and you wont get conflicts)

Is this right? Looks like I screwed up something

git help here: http://pandas.pydata.org/pandas-docs/stable/contributing.html

basically do

got rebase -i origin/master

Right, well I pretty much did that. How does the PR look?

did you `git push <username> <branchname> -f`. You have 16 commits here. You should have 1.

This is a mess, I'm just going to open a new PR for this
DOC: to_datetime outdated example
State: closed

With the new default of `errors='raise'`, the following section in the docs is not fully correctly phrased anymore: http://pandas-docs.github.io/pandas-docs-travis/timeseries.html#invalid-data. 

So I just removed that example of `to_datetime([1, '1'])` (as it now raises, which is less unexpected, and the new example already shows the three options).


Labels:
name: Docs

Comments:
@jreback BTW, as you can see in the docs, the example gives the following error message:

```
In [31]: to_datetime([1, '1'])
TypeError: invalid string coercion to datetime
```

That seems a bit a strange message?

E.g. if you only have strings, you get "Given date string not likely a datetime." (this sounds much clearer)

yeh, that original example is an odd duck as you have an actual python object which is not convvertible by default (an integer). So the error message is 'correct', but prob misleading (as its bubbling up from somewhere)
API: CategoricalIndex for value_counts
State: closed

Closes https://github.com/pydata/pandas/issues/10704

Changes `Categorical.value_counts` to return
a Series with a CategoricalIndex. Previously
the Series and an Index.

Just some random observations on the convenience of working with categoricals:
The code here could be simplified if
1. `Categorical.categories` was always a `Categoricalndex`
2. We were consistent about whether `NaN` is allowed in `.categories`

I thought that 1. would be a good idea, but I'm going back on it now. I can see how it'd be useful for `Categorical.categories` to be a more specific type like DatetimeIndex. For 2, I'd personally I'll always keep `NaN` out of my categories, but I could be missing a use case where it's needed.


Labels:
name: API Design
name: Categorical
description: Categorical Data Type

Comments:
> Categorical.categories was always a Categoricalndex

This seems slightly redundant to me. Both the categorical and the categorical index would hold an array of codes, and the categorical index would hold on to a normal index as its `categories` anyways.

> For 2, I'd personally I'll always keep NaN out of my categories, but I could be missing a use case where it's needed

Agreed.

>  Categorical.categories was always a Categoricalndex 

I would rather say: is never a CategoricalIndex. As the index should reflect the dtype of the categories, which is never a Categorical?

BTW, why would you want a CategoricalIndex (as the categories)? Is there a specific use case you have in mind where you need it?

> BTW, why would you want a CategoricalIndex

I was wanting to reindex the Series of counts with all the categories to include the zero counts, `counts.reindex(self.categories)`.

I wasn't really able to simplify the code at all. I need that `cat_idx` to get from keys _that are present_(non-zero count) and from there to all categories.

Thanks to Jeff, the change is _much_ simpler now.

lgtm.
DOC: limit error traceback to one line for expected exceptions (GH10715)
State: closed

Closes #10715

This gives you an output like

```
In [4]: s = pd.Series([1,3,5,np.nan,6,8])

In [5]: s.iloc[11]
---------------------------------------------------------------------------
IndexError: single positional indexer is out-of-bounds
```

for input

```
.. ipython:: python
   :okexcept:

   s = pd.Series([1,3,5,np.nan,6,8])
   s.iloc[10]
```

I think we would always want such a truncated traceback in the docs in cases that exceptions are expected, so I don't think it is needed to make it an extra new option. 
I only should check for cases where the actual message spans multiple lines (now it is hardcoded to give me the last line of the error traceback)

@jreback with this we don't have to use the `code-block` s in eg whatsnew for the expect behaviour of `to_datetime`


Labels:
name: Docs

Comments:
@jorisvandenbossche 

yep, actually should prob go thru the code and fix up cases where we are doing this (so that it shows the actual trace-back rather then the copy/pasted code-block). but that is prob some work; maybe make a new issue.

Looks good, although my inclination is to remove one more line of the traceback:

```
In [5]: s.iloc[11]
IndexError: single positional indexer is out-of-bounds
```

@shoyer yep, that's also certainly possible. I deliberately included the first (the line) and the last line (the message), in the idea that in this way it stands out a bit clearer as an error message. But if people prefer it without line, no problem for me and easy to change.

I prefer having just one line of the error message as @shoyer suggests.

Updated with one line (already had it done locally, but forgot to push ..)

@jorisvandenbossche doc builds are stopping since this commit
https://travis-ci.org/pydata/pandas/jobs/76637871#L1800

any thoughts? 

Hmm, stupid I didn't look at the travis builds (it builds OK locally). Thanks for noting!

It halts in enhancingperf.rst, but I did not change anything there. The only reason I could think is because there is one `okexcept` in that file. But locally that one is parsed fine (will try later on a linux machine)

gr8 thanks!