BrandonFors/pandas-github-issues huggingface dataset:
Make -NaN and -nan default NA values
State: closed

closes #5952  


Labels:

Comments:
Probably `test_default_na_values` should be brought into alignment?

@Bklyn yep on `test_default_na_values`...in theory that should fail with your change (until its fixed)....does it?

Sorry for the noob-ishness, but what is the suggested way to run the tests against not-yet-installed version of pandas (e.g. from my working copy)?  I'm getting import errors if I run the test script by hand unless I "setup.py install" first.

At any rate, the tests do _not_ fail if I don't change them, presumably because they're not trying the new -nan values.   Will fix the code and update the PR with the tests for completeness.

@Bklyn make this part of the same test (which SHOULD fail if the _NA_VALUES are not updated)

e.g.

`assert_array_equal(_NA_VALUES,pandas.io.parsers._NA_VALUES)`

This way the test have to be updated as well

@Bklyn 

`python setup.py build`

then 

`nosetests pandas/io/tests/test_parser.py`

Is there anything left for me to do on this PR?

pls add a release notes entry (put in 0.13.1 API), 

rebase and squash to 1 commit: https://github.com/pydata/pandas/wiki/Using-Git

OK, I think thats got it.

@y-p @jtratner ok by you?

fine by me.

thanks  @Bklyn 

nice little PR, and you were very responsive!!

Thanks for the hand-holding and the feedback.  You guys (and perhaps gals?) do a great job with pandas and your attention to code and documentation quality is impressive.

welcome! feel free to tackle any open issues!

Yeah, we need a gal commiter.

Well, a ninja that happens to be a gal.
FIX: extend access to `quotechar` param in `DataFrame.to_csv`
State: closed

Existing `pandas.core.format.CSVFormatter` param `quotechar` can now be
specified as keyword arg to `DataFrame.to_csv`


Labels:

Comments:
Do I need to worry about creating test(s) for this since it's passing through to an existing parameter of `pandas.core.format.CSVFormatter`?

Yikes. Travis says it broke. I follow the failed test listed in the traceback (`test_get_components_dax` in `pandas.io.tests.test_data.TestYahoo`) through to the assert statement, including downloading the target file manually. If it's the same file I retrieved, the assert should succeed. I don't see `to_csv` calls anywhere so I'm at a loss so far as to what's wrong here. 

you can force travis to rebuild that particular test (I saw that fail yesterday too...weird...but seems ok...so maybe a flaky test)

you should model some tests like other quoting type characters (prob in test_format.py or io/tests/test_parser.py).

yes write to a temp handle, then read back in and compare

It def requires a bit of care, for some reason that part is fragile

does this have an associated issue?

Most probably not--I did not file one. 

Closing, see #5414 sorry for the noise.
BUG: Possible segfault when chained indexing with an object array under numpy 1.7.1 (GH6016)
State: closed

closes #6026


Labels:

Comments:
https://groups.google.com/forum/#!topic/pydata/P3JFE3sob1U

Another demonstration that the RC only gets slightly more testing, users find these
issues very quickly after a release but many don't respond to the call-to-test for RCs.
DOC: make zipped HTML docs available GH1876
State: closed

closes #1876


Labels:

Comments:
@y-p +1 on this! 
I only have a remark: is it necesary that it is part of the `python make.py html` command? Because now, every time you build the docs locally, it will make a zip (I don't know how long it takes, so don't know if it is annoying). It could maybe also another command that is called when the docs are build for the website?

It only takes a few seconds on my box and ensures both are in sync but, point taken. 
I moved it into a new `zip_html` subcommand of make.py. 0b9570a

BTW, it does not work for me (on Windows). The removal fails, and so the rest of the command to create it is not executed (so maybe better split it up).

Also, on Windows it takes around 10s. Always has to be slower :-)

I anticipated that, hence the blanket try/except. You're welcome to do a PR or just use winzip :)

Not that important, I won't use it local, so as long as it works for the website it's OK. 
As such, the command works on windows, that's no problem. But because the file is not there yet, removing it fails and so also making the new version because it is contained in one string. 

That shouldn't be.

```
def html():
    check_build()
    if os.system('sphinx-build -P -b html -d build/doctrees '
                 'source build/html'):
        raise SystemExit("Building HTML failed.")
    try:
        # remove stale file
        os.system('cd build; rm -f html/pandas.zip;')
    except:
        pass
```

That doesn't do nothing if the file isn't there? in what way does it fail, I expected 
the `except:pass` to supress any problems, there might be some stderr noise though.

zip_html works as well. In general `rm -f no_such_file; do_stuff` will do stuff, it doesn't short-circuit.
is there something else you're suggesting?

Ah, no, I mean in the `zip_html`:

```
os.system('cd build; rm -f html/pandas.zip; zip html/pandas.zip -r -q html/* ')
```

Here the `rm` fails, so he never gets at the `zip`. The `make.py html` is working fine.

It seems something else (I assumed the above was the reason), but it's just that the zip command does work via python. `os.system('zip html/pandas.zip -r -q html/* ')`  gives `zip error: Nothing to do! (try: zip -r -q html/pandas.zip . -i html/*)` while running `zip html/pandas.zip -r -q html/*` directly in the command line works perfectly.

But let be, not important, as long as it works for the website.

That looks like a windows thing. Linux doesn't short circuit on a failed command.

I assume it is indeed not the failed command, but something with zip. But I am not going to push this to work on Windows.

so open a PR.... I won't spend time adding a zip dependency, cross-testing on windows
and adding tests, it's just a convenience.... :)

What I said, not important and I am neither going to spend time on it :-)

Hurrah for concensus!
Remove duplicated code and fix travis
State: closed

No idea why all the travis builds didn't break sooner, only numpy < 1.8
apperently forgives missing atlas at compilation time? don't know.
six errors are a red herring.

closes #6006


Labels:

Comments:
future proof :)

what? that's a thing... isn't it?

I just discovered that 'its' and "it's" are not the same thing today and I'm
feelin' fragile. don't undermine my confidence.

http://www.youtube.com/watch?v=hpiIWMWWVco
so true.

@y-p haha no underming...just jostling :)
DOC: update numpydoc to current master (commit 223df02530)
State: closed

This PR updates our local copy of numpydoc to latest upstream version (I just fully copied over that version, only left out setup.py and manifest.in). This would be a solution to the numpydoc part of issue #5221.

The intent would then be, from now on, to not touch this code, apart of pulling an updated version from numpydoc upstream from time to time. If we want to make a change, also send it upstreams to not let the two versions diverge.

**Other options** to solve this issue would be to just add numpydoc as a requirement for the doc building and removing out local option, or to add it as a git submodule. I am slightly in favor of removing our local copy, but I understood from previous comments it would be better to not have more build dependencies (and it would also tie us to a specific version).

I did build the docs with this and all seems OK. From looking at the history, I think only py3compat changes are done to these files after the initial import from Wes (@jtratner, @cpcloud you are the ones who touched these files, do you remember if you did something else than py3compat issues?)


Labels:

Comments:
Hm... It's never-ending :/

https://github.com/numpy/numpydoc/issues/8

The tests in numpydoc pass on Python 2, but with a couple of deprecation warnings and quite a few more when the [-3 flag](http://docs.python.org/dev/howto/pyporting.html#eliminate-3-warnings) is passed:

```
python -3 -m unittest discover numpydoc/tests
```

@jreback @y-p What do you think about this?

Options are:
1. Periodically updating our local copy of numpydoc, and further not touching it (**this PR does this**)
2. numpydoc as external dependency to build the docs and remove this entirely from the source
3. add it as a gitmodule (this is what numpy and scipy do), but @jtratner had some objections about this.

:+1:  for removing deps....

@jtratner what was your objection?

I like 1. stay sync'ed but don't allow external changes to break our build arbitrarily.
numpydoc is probably not moving that fast anyway.

I personally have a slight preference for option 2, but I perfectly understand the reasoning for option 1. Then, we only must have the discipline to not make changes to it (although most changes in the past were for py2/3 compat, and I hope that won't be necessary again the coming years)

So if it's 1, this can be merged.

I wouldn't say we can't change things if we need to, we should make an honest attempt
at merging upstream first IMO, is all.

afaic, merge away.

Off course, if need be, then it is. But eg the py3 compat changes were also done upstream, so that was a little bit of duplication of effort.
BUG: use astype for scipy datetime interpolation
State: closed

Closes #5975 (again), with the issue raised at https://github.com/pydata/pandas/pull/5977#issuecomment-32691271.

I'm using `astype` instead of `view` now, as Jeff suggested. I also didn't realize that `asi8` was a method on all `Index`es, and not just `DatetimeIndex`es. So instead of checking for that attr, I just check if it's a DatetimeIndex.

@y-p this should fix the doc example. It built for me anyway, but I've had trouble building the docs off my working branch.


Labels:

Comments:
this might break for a PeriodIndex
you can do

getattr(x,'is_all_dates',False)

instead of doing isinstance which will catch both types of indexes 

Thanks, I forgot about PeriodIndices. There's no need for a release notes entry right?

nope

 Thanks, added a regression test.
DOC: add guidelines for building the docs
State: closed

First draft of some guidelines on contributing to and building the docs (following #5934). Comments are very welcome.

Rendered version can be seen here: https://github.com/jorisvandenbossche/pandas/blob/doc-guide/doc/README.rst

TODO:
- [x] update section on Sphinx 1.2 now 1.2.1 is released


Labels:

Comments:
sexy!

maybe add how to show the deps: `pd.show_versions()`

I think this could form the basis for a new major doc section (in addition to a README), say 'Contributing To Pandas'?

(and also add in all of the developer links / docs as well (at some point), so that everything is in the main docs

http://pandas.pydata.org/developers.html (IOW this link would refer back to the Contributing to Pandas page itself)
which would have link for docs / code / intro...etc

Looks nice so far. I also started throwing in some notes about the things I encounter [here](https://github.com/kermit666/pandas/blob/doc-doc/doc/CONTRIBUTING.md).

@jreback Included the `show_versions()` 
And indeed, a larger doc section on ''Contributing To Pandas'' is a very good idea. But that's for later :-)

@y-p @kermit666 @phaebz Any futher comments, unclear or incorrect guidelines?

@jorisvandenbossche I think you should add as a section to the main docs

(I know you want to reorganize them at some point)....but I think they will get a lot more visibility this way 

my 2c. (you can put them near the end)

looks good to me.

Looks good to me. I left [a small comment](https://github.com/jorisvandenbossche/pandas/commit/1408467f451eba16e70fe89250bb40424b84b311) about the documentation format with a link. I don't know why didn't GitHub put it here as well. Here it is once again just in case:

```
Numpydoc allows us to format the docstrings in the pandas source code to annotate parameter and return values and present them to users in a more useful format. You can find out more about the sphinx and numpydoc format `here
 +<https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_ or simply look at some of the existing pandas documentation code to extend it in a similar manner.
```

I incorporated some of the comments. Thanks for them!

@kermit666 I think it is because you commented on the the commit. Most of the time comments are added on the "Files changed" tab at the PR, and then it certainly shows up here.

@jreback I added a section on contributing in the main documentation, but for now just included the README.rst so it is visible in the main docs. And this section can then be expanded later.

@jorisvandenbossche :+1:

bumped to 0.14.0, not a blocker.

OK, further tweaks/additions can always be made later, merging for now.

@y-p if you have a chance can you rebuild docs at some point?

docs are up.

looks great! thanks

@jorisvandenbossche contributing section looks nice!
DOC: add a couple plt.close('all') to avoid warning
State: closed

Some small doc fixes:
- added some `plt.close('all')` calls to avoid the matplotlib warning that there are more than 20 figures open.
- some style errors in release.rst. Apparantly sphinx complains about something like ```NaN``s`.because there may not be a character directly after the closing ````


Labels:

Comments:
@y-p OK to merge? (before I have to rebase release.rst with this pace of development)

Look fine, go ahead.
VBENCH: rename parser.py to parser_vb.py
State: closed

closes #5972

@dsm054, does this resolve the issue for you?


Labels:

Comments:
Yep, seems to, and I went through a full test_perf.py to make sure that the parser_vb benchmarks were still being executed.

Worked for me too. Strange that it's not fully reproducible, I haven't had this
problem. Can you report versions? would like to understand where the difference lies.

@dsm054 are you on a Mac with homebrew python?

@TomAugspurger: nope.  Had the problem on both my notebook (ubuntu 13.04) and my work box (12.04).
ENH: series rank has a percentage rank option
State: closed

closes #5971
This pull allows people to compute percentage ranks in cython for a series. I found myself computing this all the time and it will make this calculation much faster.  


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
can you add a vbench for both cases (the original and new one?)

any thoughts on 'percentage' vs. say 'per'? @jorisvandenbossche  @y-p ?

I prefer 'pct' to 'per' @jorisvandenbossche @y-p 

@MichaelWS so change to that..

done

also, I have never worked with vbench before, so I am not sure if I am doing what is intended

add a benchmark in vb_suite/groupby.py.....just copy-paste what you want to do

`./test.perf.sh -b prior_commit -t last hash_of_yours`

I am getting some odd errors. I will have to do some more research later today on it.

sqlalchemy.exc.IntegrityError: (IntegrityError) column checksum is not unique u'INSERT INTO benchmarks (checksum, name, description) VALUES (?, ?, ?)' ('ea1993ef61c3cc4e871d2cce3c5d983c', 'eval_frame_chained_cmp_python', None)

this looks fine
can u rebase and move release notes to 0.14

easy enough.  That is done.

can you reset the release.rst to master HEAD then add your change again. Their were some formatting updates which you are reverting....this should only be your 1 line change

thanks @MichaelWS !

Thanks @jreback 

@MichaelWS I think we need to have this `pct` option on DataFrame for consistency as well. Can you add?

issue is here: https://github.com/pydata/pandas/issues/6717
PERF: perf improvments in indexing with object dtypes (GH5968)
State: closed

closes #5968

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_iloc_big                               |   0.2836 |   2.8454 |   0.0997 |
```


Labels:

Comments:
That's a real boon for someone out there.

Hey now, it might be christmas again. frame_reindex is everywhere.

```
frame_dtypes                                 |   0.3270 |  66.4640 |   0.0049 |
frame_repr_wide                              |  22.0180 | 693.6400 |   0.0317 |
frame_iloc_big                               |   0.3091 |   3.5690 |   0.0866 |
frame_reindex_axis1                          | 327.7626 | 488.3307 |   0.6712 |
frame_reindex_both_axes                      |  47.7920 |  65.5576 |   0.7290 |
frame_reindex_upcast                         |  12.5876 |  16.9796 |   0.7413 |
packers_write_pickle                         |   1.0316 |   1.3046 |   0.7908 |
frame_reindex_both_axes_ix                   |  50.0793 |  60.2030 |   0.8318 |
```

Nice work! What did you use to profile this and figure out where the issue was occurring?

oh started with @y-p putting up the issue about repr of a large frame being slow 

first prob was the dtypes taking a non constant time ; 2ns was iloc 

I stepped thru it to figure out
DOC: Clarified documentation for MultiIndexes
State: closed

Clarify some quirks regarding slicing on MultiIndexes


Labels:

Comments:
One of the Travis CI builds seems to have failed due to a temporary network error:
fatal: unable to connect to github.com:
github.com: Temporary failure in name resolution

Is there some way to rerun the job that failed?

Regarding Travis, I don't think this is really a problem as this PR only touches documentation and no code, but if you want to retrigger Travis, you can reset the last commit and push --force again:

```
git commit --amend -C HEAD
git push origin yourbranch -f
```

@felixlawrence Thanks for the PR! This is indeed an area that can use some clarification.

I just made some small comments. 
And I was thinking, if you want / have the time, a small example in each docstring (from_arrays/from_tuples) would be very helpful I think. But that can also be be left for another PR.

@felixlawrence their is a button on the rhs of the screen for that rebuilt to rerun / cancel a build as well (relatively new feature on github)

@felixlawrence +1 for the examples. Maybe still another suggestion: you could add a "See also" to each docstring referring to the other. Something like this:

```
See Also
--------
MultiIndex.from_tuples : Convert list of tuples to MultiIndex
```

And lastly, before merging, could you squash to the commits to one? See https://github.com/pydata/pandas/wiki/Using-Git#fetch-and-then-rebase-interactively-to-squash-reword-fix-otherwise-change-some-commits

I added the cross-references, committed, and squashed the commits as per the link, but couldn't push the commit up to this branch (presumably because I had squashed commits locally that are present in this remote branch). What should I do?

@felixlawrence you have to force push.

@felixlawrence It seems it's OK now (the commit looks fine, and the travis failing is due to another issue (missing six)), so merging.

Thanks for the contribution!
API: add reads_json, reads_msgpack, reads_csv, reads_pickle as convience functions which accept a string
State: closed

related #5655
related #5874
closes #5924

todo:

```
[ ] update io.rst docs / whatsnew
[ ] more tests (esp pickle/csv) for passing string to ``read_csv`` or file-handle to ``reads_csv``
```


Labels:

Comments:
Not super thrilled with so many new functions... Any chance we could add a
string= kwarg to existing instead?

closing in favor of #5954 or doing nothing (better IMHO)
ENH: Improve perf of str_extract
State: closed

Building the result from a list of strings or a list of lists seems to improve performance over building the result via `arr.apply(f)`.


Labels:

Comments:
`./test_perf.sh -b master -t str-extract -r extract`

```
Invoked with :
--ncalls: 3
--repeats: 3


-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
strings_extract                              |  39.7120 | 578.9154 |   0.0686 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

```

pls add a note in release notes (u can ref this PR)
in improvements sections
looks good!

Always glad to see 10x. tx.

@unutbu ready for merge?

@jreback: Yes, ready.

thanks!
Tutorials
State: closed

added tutorials, let me know if you guys would like the output in a different format.


Labels:

Comments:
@jorisvandenbossche look ok to you?

Just made two really small comments, for the rest loooking OK!

@david Can you also squash the commits (https://github.com/pydata/pandas/wiki/Using-Git)? Then it is OK to go in.

This section of the docs will probably grow to include dozens of links,
it doens't make sense to include such ToC detail for each of them. I think
a link and a blurb would pan out better.

I am ok with the 2 existing tutorials., they are easier to navigate and sub-menus. Let's see how it goes (when the docs build on the dev site) :)

Sorry for the delay guys, I'm pretty clueless about github. How do I squash???

git fetch upstream
git rebase --interactive upstream/master
make changes and then push them back to github??

https://github.com/pydata/pandas/wiki/Using-Git

pretty close....just 's' the ones you want to squash
push with `-f` to force it to overwrite

git remote add upstream git://github.com/pydata/pandas
git fetch upstream
git rebase --interactive upstream/master

then got this msg and I am stuck?
![image](https://f.cloud.github.com/assets/2744117/1944299/34503308-7fb8-11e3-8eeb-5be0cf467485.png)

change the 'pick' to 's' for all but the first will make them all into 1 commit effecitively

save

force push

@pichonz In normal mode do `ggjV3j` then in command mode do `:'<,'>s/pick/squash/g`.

Let me know if I messed something up, this process is pretty painful.

Just takes a bit of getting used to. git can be frustrating at first. Looks like you have an empty merge there.

do

``` sh
git reset --hard HEAD~1
git push --force
```

@pichonz If you are not used to vim, you can also configure git to use eg notepad++ for the interactive rebasing (then it is easier to change the 'pick' in 'squash'). See eg here: http://stackoverflow.com/questions/1634161/how-do-i-use-notepad-or-other-with-msysgit

@pichonz , I squashed and pushed to master.  thanks for the contribution. e44d52d