BrandonFors/pandas-github-issues huggingface dataset:
DOC: Complete R interface section
State: closed

Currently, "rpy2  R interface" documentation has blank section. Added descriptions to cover required basic topics.


Labels:
name: Docs

Comments:
@sinhrks Great to work on this! (it was a little bit a stub page now)

But I am not sure of the current status of how the conversion should be preferably done. Eg see http://stackoverflow.com/questions/20630121/pandas-how-to-convert-r-dataframe-back-to-pandas, I don't know if `convert_to_r_dataframe` is still needed (with the newer rpy2)

Thanks. I've used rpy 2.3.8 and will check 2.4.0 spec.

If I look at the rpy2 docs, it seems this conversion machinery `pandas2ri` was already there from 2.3.3, although there were some changes in 2.4

@jorisvandenbossche I am ok with this

I've briefly checked `pandas2ri` of rpy2.4.0, and found current pandas conversion looks better. `pandas2ri` doesn't convert returned rpy2 `DataFrame` automatically, and may raise `ValueError` for `DatetimeIndex`. 

But it is good if we don't have to care about rpy2 instances. I think it can be achieved by simple wrapper on `pandas.rpy2.common`, and will issue a PR for this.

```
import pandas as pd
import pandas.rpy.common as com
import rpy2.robjects as ro
from rpy2.robjects import pandas2ri

pandas2ri.activate()

iris = com.load_data('iris')
ro.r.assign('iris', iris)
returned = ro.r['iris']
type(returned)
# <class 'rpy2.robjects.vectors.DataFrame'>

df = pd.DataFrame(np.random.randn(20, 5),
                  index=pd.date_range(start='2011/01/01', freq='D', periods=20))
ro.r.assign('df', df)
returned = ro.r['df']
# ValueError: Unknown numpy array type.
```

Something else, but maybe somewhat orthogonal to this PR, I think we could also mention the IPython `rmagic` in this doc page.

Note rpy2 2.4+ is now the blessed "proper" location to get the rmagic via:

```
%load_ext rpy2.ipython
```

The updated rmagic lives in that sub-package in rpy2.

@jorisvandenbossche merge when happy with this

I think it depends a bit on the outcome of  #7385?

@sinhrks ok, deferring this to outcome of #7385 

or if you want to document the current, then you can change (or submit a new one).

let's redirect to `rpy2` for conversions, see #9602
CLN/DOC: Change instances of "Pandas" to "pandas" in documentation
State: closed


Labels:
name: Docs

Comments:
I noticed what seemed to be inconsistencies in capitalization, and this PR "fixes" what I noticed. Instances in old release notes not touched, likewise for titles of talks.

If this is incorrect or unwanted, please feel free to close.

@jorisvandenbossche  ?

@jreback Do you know if there was ever a discussion on the preferred spelling/capitalization? It seems that the website (http://pandas.pydata.org/) consistently uses lower case.

But the website seems to also consistently put it in italics. Would that also be preferred for the docs?

My question was also if it is also needed to keep it lower case if it is the first word of a sentence, but it seems this is how it is done in most places.

pretty sure it should be lowercase

italics I think are just for emphasis / to break it up

so I am indifferent to italics , +1 for pandas

+1. and IMO no need for italics here.

I just find that sentences that start with a word without capital look a bit awkward, and then using italic can stress that it is not a normal word but a proper name. But OK to do this only on the landing page (as the website)

@bwignall Can you rebase this?

Rebased.

Agreed that it feels awkward to start a sentence without a capital letter. My reasoning was, if "pandas" is the proper noun, that capitalization should be used everywhere, including the beginning of a sentence. Of course, happy to revert those if that is incorrect.

OK, but for now the rule is then 'always and everywhere lowercase". So this is OK, merging. Thanks!
BUG: Index.delete doesnt preserve name and other attrs
State: closed

Made `Index.delete` to preserve `name`, `tz` and `freq` attributes.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks

Ok, putting this comment in the right place now (was on #7299):

I just pulled this change in and it threw me for a loop until I figured out what was going on (and hopefully I understand it correctly). I was working on fixing some plotting bugs like #2960 and this actually changes the behavior of the setup case provided in that issue. It used to be that dropping items from a regular timeseries would make that timeseries "irregular" (by dropping the freq), making it plot differently, etc.

I'm fine with this new behavior but was the old behavior there for a reason? I had gotten used to it.

have to revisit this (in 7299)

because adding/deleting a stamp can change the freq (eg make it none); though it could also not change it if say deleting from the end 

I think the behavior was to always make it reinfer - so not sure it should actually different

do u have an example?

Sure, here's an example. The last case (inspired by #2960) differs. Maybe that's fine?

(I thought it might be more far-reaching but apparently not.)

On master:

```
In [14]: ts = pd.Series(index=pd.date_range('2000-01-01', periods=10, freq='D'))

In [15]: ts[:5].index
Out[15]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-01, ..., 2000-01-05]
Length: 5, Freq: D, Timezone: None

In [16]: ts[[1,2,3]].index
Out[16]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-02, ..., 2000-01-04]
Length: 3, Freq: None, Timezone: None

In [17]: ts.drop(ts.index[[5,6,7]]).index
Out[17]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-01, ..., 2000-01-10]
Length: 7, Freq: D, Timezone: None
```

On 0.14.0:

```
In [1]: ts = pd.Series(index=pd.date_range('2000-01-01', periods=10, freq='D'))

In [2]: ts[:5].index
Out[2]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-01, ..., 2000-01-05]
Length: 5, Freq: D, Timezone: None

In [3]: ts[[1,2,3]].index
Out[3]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-02, ..., 2000-01-04]
Length: 3, Freq: None, Timezone: None

In [4]: ts.drop(ts.index[[5,6,7]]).index
Out[4]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-01, ..., 2000-01-10]
Length: 7, Freq: None, Timezone: None
```
TST: fix intermittent dict_complex test failures
State: closed

`test_dict_complex` and `test_dict_numpy_complex` would give occasional test
failures on Python 3.4.  The test was written to depend on the order of values
from `dict.values()`; remove that dependency.

Closes gh-7293.


Labels:

Comments:
Good point - how about this version?

yeh these tests were ripped from the numpy-msgpack impl 

thanks !

nice!
CLN/TST remove compat.scipy
State: closed

re-branch of https://github.com/pydata/pandas/pull/7253#issue-34432164

-Move `percentileofscore` to `stats.misc`; Update `percentileRank` to reflect the move.
-Change testing references to `compat.scipy.rankdata()` , to use `scipy.stats.rankdata` directly
-Delete compat.scipy 


Labels:

Comments:
Does this mean you can no longer use `.rank` without scipy? (The whole point of that compat file was so that scipy wasn't a dependancy, right?)

Ah ok, this is just in tests. Cool.

yeh I think was just to check vs scipy (and maybe it wasn't included in testing suite at that point)

@hayd, `_skip_if_no_scipy` should probably be in there. I'll push up the change. 

Sorry guys... I'm having some git-problems...

It looks like you should remove the last commit (the merge one): http://stackoverflow.com/a/6866485/1240268

(I would first check that's the case with `git log` before you nuke it...)

Then force push back to this PR: `git push origin CompatScipy -f`.

I think you forgot to `import nose`, see the [allowed failure](https://travis-ci.org/pydata/pandas/jobs/26478521).

```
ERROR: pandas.tests.test_tseries.test_rank
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.6/lib/python2.7/site-packages/nose-1.3.3-py2.7.egg/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/virtualenv/python2.7.6/lib/python2.7/site-packages/pandas/tests/test_tseries.py", line 344, in test_rank
    _skip_if_no_scipy()
  File "/home/travis/virtualenv/python2.7.6/lib/python2.7/site-packages/pandas/tests/test_tseries.py", line 16, in _skip_if_no_scipy
    raise nose.SkipTest("scipy not installed")
NameError: global name 'nose' is not defined`
```

Ah, take that back with the "just remove last commit", I see one commit is there twice. I would either:
- create a new branch (from this one) and squish the commits (`git rebase -i HEAD~4`).
- create a new branch (from master) and cherry-pick the commits you want (`git cherry-pick commit_sha`).

Then push back here: `git push origin branch_name: CompatScipy -f`.

@hayd Thank you for the bail-out. My local git is pretty well munged at this point... What is the best way to nuke it?  I've saved the "correct" copies of all affected files elsewhere so I can just drop them in. Should I just nuke the branches and do a fresh PR, or is there better way?

I wouldn't nuke it just yet... create a new branch off master (assuming this branch was from there) and then drop them in or cherry-pick. You can then force push to this PR (using the code above), no need to nuke locally til it's successfully up here.

That is:

```
git checkout master
git checkout -b CompatScipy2
# do the stuff (either cherry-pick or drop in the files and commit)
# push your local branch to this PR with
git push origin CompatScipy2:CompatScipy -f
```

If it's up here ok, then you can go crazy deleting branches.

@hayd Thanks for the bailout!!

np! This looks great.
BUG/CI/TST: fix format strings
State: closed

Also test against an unsupported version of openpyxl (2.0.3)

closes #7284


Labels:
name: Bug
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO Excel
description: read_excel, to_excel
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
repro'd the error so that's good:

https://travis-ci.org/cpcloud/pandas/jobs/26397310

does the warning happen of openpyxl is not installed at all? (or does this not get that far)
do we test if it's not installed but running the tests anyhow (need to assert that no warning is printed)

@jreback yes warning does happen on import of pandas ... thx for pointing that out

should this warn on import? or when openpyxl is actually used?

it really should warn only when used

in that case it should throw an error because it doesn't really work with incompatible versions ...

either warn at import or raise on use

we do this for html so it's not unprecedented

yeah maybe just that

well that was fun

is their a build that doesn't have openpyxl installed at all?

no they all have it

could remove from one of the py3 builds

yep take out the 3.2 maybe

just to make sure it fails (and also does not fall back to another engine - u have to set explicitly iirc)

@jreback ok here after squash?

yep though maybe add a note in whatsnew 

good call

ok...merge when ready

https://travis-ci.org/pydata/pandas/jobs/26564920

so i think you need to `_skip_if_no_openpyxl` in this test (before calling `openpyxl_compat.is_compat()`

numpy 1.9_master tests build doesn't have openpyxl installed at all (nor ANY excel stuff)

Hm ok must be happening in more than one place bc I fixed one of those already. 

Oh on

Ok
ENH/TST: add anonymous reading of s3 for public buckets
State: closed

closes #7246


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
need to add to requirements for the ones you want to check (maybe just 2.7), test the non-implemented error in the rest?

ah right thx ... 

@jreback ok with this?

will squash 

one thing is that i use a symlink to avoid copying tips.csv ... will that work on windows?

worst case i can just copy tips.csv

looks ok
no idea about the symlink
FIX DataFrame diff with timedelta 
State: closed

fixes #4533

_need to fix test in numpy 1.6..._


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Timedelta
description: Timedelta data type

Comments:
hmm seems a bit special casey

true, perhaps better to change the dtype of arr_out / view, but couldn't get that to work...

I think you need to set the dtype of `out_arr` in the if block (e.g. if its float/int/datetime64/timedelta64) as it needs to be `i8` (then you take a view before you return it, so need to remember to do that)

e.g. `return out_arr.view('timedelta64[ns]')`

@hayd ?

@hayd ?

@hayd ?

@jreback Did you think this was too special casey? (will change the view dtype and see if this passes).

@hayd it might be, but not sure how can avoid this, `timedelta` is ITSELF special casey (integer and datelike at the same time)

@jreback ok cool, in general it's good to be uncomfortable/point out/whine about special casey-ness. :)

closing this as picking it up in #8184 

thanks @jreback, sorry for dropping the ball on a few PRs.
BUG: hist raises TypeError when df contains non numeric column
State: closed

`DataFrame.hist` without `by` keyword raises `TypeError` when it contains non-numeric column.

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import pandas.util.testing as tm

n=100
gender = tm.choice(['Male', 'Female'], size=n)
classroom = tm.choice(['A', 'B', 'C'], size=n)
single = tm.choice(['S'], size=n)

df = pd.DataFrame({'gender': gender,
                   'classroom': classroom,
                   'height': np.random.normal(66, 4, size=n),
                   'weight': np.random.normal(161, 32, size=n),
                   'category': np.random.randint(4, size=n)})

df.hist(by='gender')
# This works, even though 'classroom' (str) column in still contained

df.hist()
# TypeError: cannot concatenate 'str' and 'float' objects
```


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@TomAugspurger ok by you?

Merged via https://github.com/pydata/pandas/commit/fa0f78fd9989d4f68b0a52ed505a4c7a7df08f55

Thanks.

@TomAugspurger I just uploaded mac osx wheels to PyPi. Can you try to install and see if it works?

Tested on 2.7 and 3.3. Both of those worked.

excellent!

This build [failed on Travis](https://travis-ci.org/pydata/pandas/builds/26411039), but the failure seems to be unrelated. I restarted the build.

@TomAugspurger @jreback Thanks to merge. The failure seems to be caused by unrelated part. I've created a issue #7289.
DOC/RLS: remove duplicate content in release.rst and move remaining to whatsnew
State: closed

Related to discussion on mailing list https://mail.python.org/pipermail/pandas-dev/2014-May/000272.html

So I gave it a try, moved all content of release.rst to the whatsnew file (main chunk were the bug fixes, and for the rest some api changes and enhancements that were not listed in whatsnew). All content that was in release is not in whatsnew.

Can you see what you think of this? 


Labels:
name: Docs

Comments:
looks great!

also, can you build the final docs and either e-mail or put them on the server?

(can always rebuild them if any issues). But I need them to provide links for the announcement doc.

thanks

@jorisvandenbossche actually don't need the docs built for the ANN. let's get this in and I'll take a look at the built docs final time today.

I pushed some small fixes. Rest of the docs are building fine, I will send you later this evening or tomorrow

gr8 thanks!
ENH add sample #2419
State: closed

fixes #2419

Hmmm `np.random.choice` not available on numpy < 1.7.


Labels:
name: Enhancement

Comments:
closing for now. @hayd if you want to update at some point, ok,.

Just wrote a little utility function to do this, so I may pick this up.

``` python
def sample_n(df, size, replace=False, weight=None, seed=None):
    rs = np.random.RandomState(seed)
    locs = rs.choice(df.shape[0], size=size, replace=replace, p=weight)
    return df.take(locs, axis=0)
```
FIX resample with fill_method and how #2073
State: closed

fixes #2073


Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Resample
description: resample method

Comments:
just needs a whatsnew note otherwise looks good

@hayd status on this? i can take it if you want ... pretty much done. passing on travis

If you've done it push it! :)

ok closing in favor of #7341 (basically just your commits but re run)

Great, thanks!

confused at the travis: IIRC travis passed, then I squished in some release notes, then it failed.

looks like a random failure ... since you didn't really touch any of the period index stuff you're just calling fillna which shouldn't change any keys ...
BF: reading stata files - unpack read value describing stored byte order
State: closed

Otherwise comparison would always result in "low endian" .  Seems to resolve all the gory stata files reading tests on sparc at least 
closes #5781 


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
whoosh! thanks seems simple enough

thanks @yarikoptic 

you are welcome

Never give up -- never give in! ;)

On Thu, 29 May 2014, jreback wrote:

>    thanks [1]@yarikoptic
> 
>    —
>    Reply to this email directly or [2]view it on GitHub.
> 
> References
> 
>    Visible links
> 1. https://github.com/yarikoptic
> 2. https://github.com/pydata/pandas/pull/7272#issuecomment-44521545

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

That was a simple fix.  In principle someone should probably test the .dta files created on a big endian platform in Stata. I have tested the files created in the test suite in Stata, but only on LE.

Hard since most commodity hardware is using x86 which is little endian. Sparc I think is only on specific sun systems which are not feasible for any of us to purchase. That said sometimes these builds do find bugs. 

yep the debian guys run all of the builds on lots of hardware

On Fri, 30 May 2014, Phillip Cloud wrote:

>    Sparc I think is only on specific sun systems which are not feasible for
>    any of us to purchase. 

FWIW: used sparcs are actually more than affordable.  companies simply get rid
of them after a year or two pass the service contract expiring. And
since they are not that popular as a commodity x86 -- they are quite
cheap

look at
http://www.ebay.com/sch/i.html?_trksid=p2050601.m570.l1313&_nkw=sparc&_sacat=0&_from=R40
for a hundred or two you can get a reasonable sparc box

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Interesting. Thanks @yarikoptic

I am sorry for ruining your dream @cpcloud that you had a very wealthy cyber-friend -- my sparcs are recycled 2000 and 2005 boxes too ;)

You must use Raid for debugging those :)

>    You must use Raid for debugging those :)
> whatever works... ;)

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
ENH: check for __array__ instead of ndarray subclasses when creating an Index
State: closed

This allows custom ndarray-like objects which aren't actual ndarrays
to be smoothly cast to a pandas.Index:
https://github.com/pydata/pandas/issues/5460#issuecomment-44474502


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
Could someone kindly retrigger the Travis build?

I'm not sure exactly what happened, but I think the failure is spurious: the build passed when Travis ran on the branch in my fork:
https://travis-ci.org/shoyer/pandas/builds/26263713

rebase on master and push again (this was just fixed in master).

as an aside, you can `git commit -C HEAD --amend` and force push to resend the build

OK, Travis passed after the rebase.

gr8. we will need a release note (but 0.14.1 not up yet....check back in 2 days)

No chance I could sneak this in 0.14.0? :)

On Thu, May 29, 2014 at 10:58 AM, jreback notifications@github.com wrote:

> gr8. we will need a release note (but 0.14.1 not up yet....check back in 2
> days)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7270#issuecomment-44563042.

ok no biggie (dont' worry about the release note then)....that's changing (all going to be in whatsnew from now on)
added  functionality to allow negative loc values to work with Index.insert
State: closed

closes #7256 


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype

Comments:
needs tests

Added testing @jreback to the class `test_index.TestIndex`, ... it looked a little daunting at first, but I think I ended up getting it

To clarify, do you mean `Index.insert` onto an empty `Index`?

yep (for all three cases) (and if loc > 1 or < -1 then it will raise a IndexError); tests are prob their for that though

Thanks for clarifying, making those changes now.

Added test cases for:

[x] null
[x]+/- 
[x] neq (0,-1)
[x] loc = (0,-1)  

https://github.com/pydata/pandas/issues/7256

You still need a +1 in order to get the indexing to work  properly, or else you get the same "off by 1 error."  I'm more than happy to make it `if loc < 0: loc += len(_self) + 1`

ok...looks good.pls squash down to 1 commit

I **think** I did what you were asking for... I've never squashed commits down to a single, but it's under commit `471c8f5`.

see here: https://github.com/pydata/pandas/wiki/Using-Git

you `git rebase -i master`, squash everything (or force together)

then force push

@jreback, we good, now?

you still have lots of commits.

```
git rebase -i upstream/master
```

then s/f

```
git push yourfork thisbranchname -f
```

doesn't work for you?

That was it, thanks.

[last commit](https://github.com/benjaminmgross/pandas/commit/adb1a453ba651e6f9faaca602ef5fcf3b01f51f9)

merged via https://github.com/pydata/pandas/commit/962ca2c5837151e36decb23f7835d4f8771aaa47

thanks!