BrandonFors/pandas-github-issues huggingface dataset:
improved error message for invalid chart types
State: closed

Closes #9400

`pd.Series([2,3,4]).plot("quiver")` now gives an improved error message:

```
Users/ch/miniconda/envs/pd_dev34/lib/python3.4/site-packages/pandas/tools/plotting.py in _plot(data, x, y, subplots, ax, kind, **kwds)
   2269         klass = _plot_klass[kind]
   2270     else:
-> 2271         raise ValueError(invalid_ct_fmtstr % kind)
   2272 
   2273     from pandas import DataFrame

ValueError: 'quiver' is not a valid chart type
```

instead of:

```
/Users/ch/miniconda/envs/sci33/lib/python3.3/site-packages/pandas/tools/plotting.py in _plot(data, x, y, subplots, ax, kind, **kwds)
   2267         klass = _plot_klass[kind]
   2268     else:
-> 2269         raise ValueError('Invalid chart type given %s' % kind)
   2270 
   2271     from pandas import DataFrame

ValueError: Invalid chart type given quiver
```


Labels:
name: Visualization
description: plotting
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Can you check the new error messages? If they're ok, I would squash my commits.

This looks good to me. @jorisvandenbossche ?

@cel4 Just a small question, can you squash the two commits into one?

@cel4 Looking good, thanks! I'll merge when travis is green
BUG: setup.py does not allow equal filenames in *different* directories
State: closed

Is it possible to write tests for setup.py?
## Setup

```
touch pandas/period.pyx
```

and put this in your `setup.py`'s ext_data:

``` python
    period=dict(pyxfile='period'),
```
## Before PR

``` bash
python setup.py build_ext --inplace
rm -rf pandas/period.c
python setup.py build_ext --inplace
python setup.py clean 
ls -l pandas/period.c
# -rw-rw-r-- 1 brandon brandon 48253 Feb  4 15:27 pandas/period.c
```

`period.c` still exists after cleaning.
## After PR

``` bash
python setup.py build_ext --inplace
rm -rf pandas/period.c
python setup.py build_ext --inplace
python setup.py clean
ls -l pandas/period.c   
# ls: cannot access pandas/period.c: No such file or directory
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Build
description: Library building on various platforms

Comments:
you _could_ write tests for `setup.py` in that you could assert existance of certain files known to be there. It would guard against a file being removed that is known (e.g. `period.c` which _should_ be there), but, if a new file were added then it couldn't assert for that (e.g. say you added a module which add `foo.c`), so would be a bit tricky to keep the tests in sync.

So kind of +/- 0 on this.

@jreback gotcha. thanks for the input on testing.

@blbradley thanks for this

@jreback That was the easy one. :smirk:
test sql table name
State: closed

Closes #9393 

simple test for tables names that need to be quoted.
This is related to issue [here](https://github.com/pydata/pandas/issues/9393) that is fixed in current master.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@dashesy The sql tests are here: https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_sql.py

yes, tests are looking good now! (and for fixing the test :-))

When do you get the `AttributeError: '_TestSQLApi' object has no attribute 'connect'` errors? What do you run exactly?

Further, could you squash the commits into one (for merging the PR)? (`git rebase -i upstream/master`) If you have questions about it, just ask!

@jorisvandenbossche  I test in Eclipse/PyDev by running the module as a test unit.

@dashesy Thanks for the test!

When working on the sql code, I just run the `test_sql.py` file (there is a main function in the end) and that works for me.
Fix Panel.to_excel for unicode strings in axis 0
State: closed

closes issue #9404


Labels:
name: Unicode
description: Unicode strings
name: IO Excel
description: read_excel, to_excel

Comments:
- pls add a test for this
- pls add a release note in v0.16.0.txt

this needs a test

closing as stale, but if you would like to reopen and fixup pls do.
ENH: - Fixed column_format propagation to DataFrameFormatter in to_latex
State: closed

closes #9402  


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
can you add/update a test in `test_format` to use this paramater (at least as a smoke test).

can you rebase this

closing as stale, but if you would like to reopen and fixup pls do.
TST/DOC: Add procedure for TestPickle
State: closed

Added procedure to `TestPickle`. Based on #9291, I think updating `setup.py` is likely to be ommitted.


Labels:
name: Docs

Comments:
yep. Is there any way to have setup automatically pick up all of the directories under a tree in package_data ?

@jreback Yeah it's better. Modified both `setup.py` and `test_pickle.py` to test all files under "legacy_pickle".

OK. I understand that to make `nose` to find dynamically added test methods, we have to use test generators. Thus, I changed the `TestPickle` not to inherit `unittest.TestCase`. If there is a better way, please lmk.
http://stackoverflow.com/questions/6689537/nose-test-generators-inside-class

When execute the test, we will see:

```
__main__.TestPickle.test_pickles('0.10.1',) ... ok
__main__.TestPickle.test_pickles('0.11.0',) ... ok
__main__.TestPickle.test_pickles('0.12.0',) ... ok
__main__.TestPickle.test_pickles('0.13.0',) ... ok
__main__.TestPickle.test_pickles('0.14.0',) ... ok
__main__.TestPickle.test_pickles('0.14.1',) ... ok
__main__.TestPickle.test_pickles('0.15.0',) ... ok
__main__.TestPickle.test_pickles('0.15.2',) ... ok
__main__.TestPickle.test_round_trip_current ... ok

----------------------------------------------------------------------
Ran 9 tests in 1.359s

OK
```

@sinhrks ok 

a possible 'soln' to this is to make make all of the methods in tm.TestCase (currently) into a MixIn (and include it there, so nothing changes). And then you can use the MixIn here.

@jreback Based on #8923, better to use methods under `tm...` without creating mixin? I've modified test cases to use them.

thanks for this!
MAINT: get rid of some compiler warnings
State: closed

- uninitialized pointer
- unused variables
- unused function
- ISO C prototypes
- `cimport *`
- useless cast before `snprintf`


Labels:
name: Build
description: Library building on various platforms

Comments:
Thanks for looking into this.

What compiler are you using? I may have a chance to try it out with clang.

GCC 4.7.

can you rebase this?

@jreback done.

can you quantify I guess how many warnings this removed? (as obviously still lots)

I don't know, four, five? The code has moved on since I wrote this patch and I have no idea how many new warnings were introduced.

this looks ok, but to be sure, can you rebase and push again? ping when green.

Green!

thanks!
ENH: Add StringMethod.find and rfind
State: closed

Derived from #9111.


Labels:
name: API Design
name: Strings
description: String extension data type and string data

Comments:
Thanks, fixed.

OK, looks good to me.

@sinhrks I added some documentation comments. Further:
- I think you may group all the StringMethods enhancement in a section under 'new features' (as it starts to become a large list)
- Maybe we should start adding `versionadded` directives (see [here](http://sphinx-doc.org/markup/para.html#directive-versionadded)) to newly added functions? We don't do this consistently right now, but I think we should do that more.

and another:
- maybe add a 'Notes' section saying it is equivalent to the standard library's `str.find` (I even think you can link to that using `:func:`str.find``

Modified once.
- About release note, let me do once finish #9111 (added as AI).
- Adding `Notes` is duplicated with the "Method Summary" in `text.rst`. How about adding `Notes` in each docstring, and change "Method Summary" section to a simple link to "String handling" section in `api.rst`. 

And on the note, I don't think that is a problem that is would be duplicated (that is the point of having both docstrings and narrative docs, there will always be some duplication). 

If the table in text.rst can be replaced by an autosummary table is separate issue I think. Positive side is that it would add the explanation manually, but at the moment I like more how the table looks in text.rst (better overview) than how it is on the api pages. 

It is maybe not necessary to add it in a Note, just adding 'equivalent to standard library's str.find' in the explanation in the beginning of the docstring is also good.

Maybe my word choice is inappropriate. What I'm caring is uniformity of styles and descriptions. What I expect is all the docstring should have similar format, and sufficient information in a single place.

In this respect, current doc should be revised in separate PR to cover:
- All the docstring should provide "better overview" and this should be used both in "text.rst" and "api.rst".
- Add descriptions for standard `string` methods (not only `find` and `rfind`). 

This is one thing, and added a description for this PR.

thanks @sinhrks
bug in groupby when key space exceeds int64 bounds
State: closed

closes https://github.com/pydata/pandas/issues/9096

also improves performance when there is `int64` overflow; complete groupby benchmarks [here](https://gist.github.com/behzadnouri/e142fbd65f41357a7360).

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_int64_overflow                       | 637.6967 | 3021.1190 |   0.2111 |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [d6e3cb7] : bug in groupby when key space exceeds int64 bounds
Base   [391f46a] : BUG: allow for empty SparseSeries SparsePanel constructors (GH9272)
```


Labels:
name: Bug
name: Groupby

Comments:
@behzadnouri this looks gr8! thanks
PERF: nunique perf improved by using len(unique) rather than value_counts
State: closed

Before:

```
In [1]: s=pandas.Series(range(100)*1000)
In [2]: %timeit s.nunique()
1000 loops, best of 3: 1.43 ms per loop
```

After:

```
In [1]: s=pandas.Series(range(100)*1000)
In [4]: %timeit s.nunique()
1000 loops, best of 3: 440 µs per loop
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Can you also time this for an example with NaNs ?

It would be nice to also formalize this timing with a new vbench.

Here are the examples with NaNs:

```
In [8]: import pandas

In [9]: s=pandas.Series(range(100)*1000)

In [10]: dropna=True

In [11]: %timeit len(s.value_counts(dropna=dropna))
1000 loops, best of 3: 1.6 ms per loop

In [12]: %timeit s.nunique(dropna)
1000 loops, best of 3: 450 µs per loop

In [13]: dropna=False

In [14]: %timeit len(s.value_counts(dropna=dropna))
1000 loops, best of 3: 1.61 ms per loop

In [15]: %timeit s.nunique(dropna)
1000 loops, best of 3: 441 µs per loop

In [16]: s=pandas.Series((range(100)+[None])*1000)

In [17]: dropna=True

In [18]: %timeit len(s.value_counts(dropna=dropna))
100 loops, best of 3: 8.61 ms per loop

In [19]: %timeit s.nunique(dropna)
1000 loops, best of 3: 1.22 ms per loop

In [20]: dropna=False

In [21]: %timeit len(s.value_counts(dropna=dropna))
100 loops, best of 3: 9.24 ms per loop

In [22]: %timeit s.nunique(dropna)
1000 loops, best of 3: 1.2 ms per loop
```

Could you give me an example for the vbench? Should it be in the pandas project or in the vbench project?

I did not rebase with the last version of master. And this had been done in ff124f9efec602ad6de552a0ba41428d69615612
I close it

@gtnx sorry about that I forgot this _was_ fixed (and the issue was orphaned).
BUG: Fix buffer overflows in tokenizer.c that caused python to segfault with certain
State: closed

closes #9205 


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
whoosh, quite a number of changes for a 'simple' bug.....

there was some pruning of routines that didn't seem to be used anymore and some changes to the way pointers were reallocated and freed while investigating the problem.  the heart of the fix is checking lengths against the buffer caps as items are added to the buffers.    i thought the extra checking might slow things down slightly, but vbench claims the tokenizer branch is as fast as upstream/master.  i'll take a look at the other csv issues you pointed me to earlier

@selasley hahh.. wasn't being critical. I am very glad you refactored. Sometimes just needs to be done! thanks.

will have to test this on windows.

:)  I was a bit nervous about changing so many lines of other people's code.  Thanks for checking on windows.  I don't have access to a windows machine at home or work to do any testing.
I fixed the typo in the whatsnew document and pushed the changes

ok, this looks good on windows.

can you do a vbench run on this? just to confirm nothing has changed? 

ping when done and we'll get this in.

merged via https://github.com/pydata/pandas/commit/c6c9c0bfc10e24bdd0e4d32f7a8faf7fbc2294b2

thanks!
CLN: Bring pandas up to date with pandas-datareader
State: closed

Per: pydata/pandas-datareader#15

Fixes #9010, #9026 


Labels:

Comments:
thanks @dstephens99 

if there are any other corrections to bring these in sync. pls lmk asap.

@jreback Just diff'ed the pandas-datareader data file with the one in pandas - they are the same, everything is up to date.
Deprecating the trellis rplot module (GH3445)
State: closed

Closes #3445 

---

Start for #3445. Then the current examples should get a seaborn or ggplot alternative.

Questions: 
- explicitely refer to seaborn and ggplot as alternatives? (as there may be other packages that could feel neglected that way?)
- I now raise the warning on import of the module (`import pandas.tools.rplot as rplot`). The only thing is that the warning will be raised in the tests this way


Labels:
name: Visualization
description: plotting
name: Deprecate
description: Functionality to remove in pandas

Comments:
> @TomAugspurger Did you already have some examples of how to do it in seaborn? I vaguely can remember you posted it once in an issue? 

@jorisvandenbossche I redid all the examples using seaborn. A quick search didn't turn it up, but I'll dig for it later.

There are ways to both suppress and check for warnings in the tests, we have a couple of examples of it in mpl's test suite.

I'm not sure ggplot is ready for prime time. I would probably only recommend Seaborn at this point.

@shoyer yes, I could agree, as I noticed while trying to convert some of the examples to gpplot this was not easy (some of the functionality of rplot is not yet included in ggplot). I will update the PR

I agree too: ggplot currently doesn't handle categorical data very well if faceting is used (actually that was my motivation for working on categorical data in pandas :-) ). 

OK, I will remove ggplot for now from the explanation, and if @TomAugspurger finds his seaborn examples, we can add that. But @JanSchulz, if you want to add some ggplot examples when there is some progress, certainly do!

@jorisvandenbossche @TomAugspurger let's do this for 0.16.0 (pls add a refernce in #6581) as well

I made some examples of how to do it with seaborn, for each example in de rplot docs now: http://nbviewer.ipython.org/gist/jorisvandenbossche/103876bff4a5f42c9933#

@TomAugspurger Can you have a looks if this looks good? Or if there are better ways? (only for the last example (4 variables) I did not found a fully equivalent seaborn-way of doing it)

@jreback For the actual PR, I assume adding seaborn as a doc dependency is not that handy, so I better include all the figures as images?

Joris, I remember talking to Michael about that last plot and it was agreed
that seaborn didn't have a way to do that plot directly. It was also agreed
that this is a good thing, since that chart is a mess :) Thanks for redoing
this

I fully agree about that mess :-)

The rest is looking OK?

This looks great to me!

@jorisvandenbossche I think just need a release note in the deprecations sections

@jreback ah yes!
Add various display types for pivot table such as Excel
State: closed

Excel offers several types of values display for a pivot table (ratio of row, ratio of column, ratio of total, difference with, cumulated sum ...)
This ticket aims to implement this option in the pivot table API. I implemented the three types cited above for a start.
I chose "display_value" for the argument name. One can find too generic, so i'm opened to suggestions.


Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I think this could be better in the cookbook. Closing. But if you want to put an example there would be gr8.

If you want to display values as ratio of column/row/table total like Excel/Calc allows, here's a piece of code which will do the job:

```
if display_value == "col_ratio":
    # Display values as a ratio of column total
    table = table.div(table.sum(axis=0), axis=1)
elif display_value == "row_ratio":
    # Display values as a ratio of row total
    table = table.div(table.sum(axis=1), axis=0)
elif display_value == "total_ratio":
    # Display values as a ratio of table total
    table = table.div(table.sum().sum())
```

Indeed, I think this would be a nice example to add to the docs.

@gtnx In general, if you want to prevent the fact that we close a PR, it's better to first open an issue asking if such an enhancment would be desired.

You're right. I realized it after i did the dev. I will do that next time
ENH: StringMethods now supports ljust and rjust
State: closed

Derived from #9111.
- Add `StringMethods.ljust` and `StringMethods.rjust` to be consistent with standard `str`.
- Add `fillchar` option to `StringMethods.pad` and `StringMethods.center` to be consistent with standard `str.center`
  - https://docs.python.org/2/library/string.html#string.center


Labels:
name: API Design
name: Strings
description: String extension data type and string data

Comments:
this needs to go after  #9322 (and might need a rebase after that) as the docs are changing a bit)

Just merged #9322, so this needs a rebase now.

thanks @sinhrks
Changed uin8 to uint8 in response to issue #9266
State: closed

closes #9266 


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks, this needs:
1. a test case which breaks before this change (you can use the example from #9266; add it to `pandas/io/tests/test_parsers.py`)
2. an addition the "bug fixes" section of `docs/sources/whatsnew/0.16.0.txt` (refer to the original issue)

This is my first PR attempt and I don't understand this error message.  Did I do something wrong in the test function I added?

@CoolRanch29 I rebase and push your branch up here. pls review and update this PR when you can
https://github.com/jreback/pandas/tree/CoolRanch29-master

I am not sure what you are trying to compare against here (e.g. what should the expected frame look like?)

Okay, I added an expected frame to test again.  The problem before is that pd.read_fwf wasn't working at all - an exception was thrown when trying to read the test case.  I'm still concerned that in the resulting frame, the column 'dst' is of type 'object' when using read_fwf and of type 'float64' when created by DataFrame().

whats new is now available for 0.16.1 https://github.com/pydata/pandas/commit/5ebf521eac528f5a4b772e4677f70b689b8b1826

can you rebase this?

I'm pretty sure I just rebased.  I'm sorry if this is causing a headache for you, I'm sure I'm doing some things stupidly.  I've gotten a lot of benefit out of using pandas and so I'm trying to contribute but this is my first time contributing so I'm having to learn a lot about both the pandas code and github.  

couple of style things. pls add a release note in v0.16.1 as well.

We'd like to merge this. Pls address the comments from above. (and needs a rebase). see [here](http://pandas-docs.github.io/pandas-docs-travis/contributing.html) for help on this.

@CoolRanch29 can you update

@CoolRanch29 want to update?

Okay, I believe I just rebased.

On Wed, May 20, 2015 at 7:06 AM, jreback notifications@github.com wrote:

> @CoolRanch29 https://github.com/CoolRanch29 want to update?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9350#issuecomment-103844130.

See here for rebasing help. Pls squash to a single commit as well. http://pandas.pydata.org/pandas-docs/stable/contributing.html

can you rebase/squash
Fix gbq client to only return results when jobCompleted is True.
State: closed

When polling for a long-running gbq job to determine if it is complete, we should only return results once `query_results['jobCompleted']` is `True`, not just when the `jobCompleted` key exists.  Otherwise, the gbq client thinks results exist and it will start attempting to parse the results, leading to a weird KeyError:

```
/Library/Python/2.7/site-packages/pandas/io/gbq.pyc in read_gbq(query, project_id, index_col, col_order, reauth)
    368▓
    369     connector = GbqConnector(project_id, reauth = reauth)
--> 370     schema, pages = connector.run_query(query)
    371     dataframe_list = []
    372     while len(pages) > 0:

/Library/Python/2.7/site-packages/pandas/io/gbq.pyc in run_query(self, query)
    192                             jobId=job_reference['jobId']).execute()
    193▓
--> 194         total_rows = int(query_reply['totalRows'])
    195         result_pages = list()
    196         seen_page_tokens = list()

KeyError: 'totalRows'
```

This simple patch accounts for the case where `query_results['jobCompleted']` is `False`.


Labels:

Comments:
this is addressed in #8728 which is pending some more testing
Add lag parameter to autocorrelation
State: closed

Add lag parameter to autocorrelation, default to lag-1 autocorrelation
so existing code will work unchanged.

This is effectively reopening issue #9192


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
this is ok, @jorisvandenbossche ?

merged via https://github.com/pydata/pandas/commit/d349adc06974f190dff05fee87e876c76f423300

thanks!