BrandonFors/pandas-github-issues huggingface dataset:
EHN: Allow load_data to load problematic R datasets
State: closed

TST: Move tests from rpy/common.py to tests/test_rpy.py
TST: Add tests to demonstrate the enhancements made to rpy/common.py.
DOC: Add explanation to doc/source/release.rst


Labels:

Comments:
This is related to #5036, which I unwittingly closed while trying to clean up my commit history.

@unutbu looks fine...and since you are the one to test it, good to go?

actually....do you want to put a mention (maybe code-block example?) in v0.13.0?

Yes, I think it is good to go.

(Assuming the change to v.0.13.0.txt is acceptable).

@unutbu can you move the test file into a subdirectory of rpy?
so it would go into `rpy/tests/<your_filename_here>.py`

ready to go on this?

Need to add init file to test folder, right? (maybe I'm just missing it) 

I hope this isn't too many commits. If so, I could delete this branch and start a new one with a shorter history. Please tell me if there is a better way.

can u rebase and squash?

no need to delete 

https://github.com/pydata/pandas/wiki/Using-Git

@jtratner @cpcloud merge?

let me test this out

needs a rebase ... there's probably a merge conflict with release or v0.13.0.txt

looks good on my end ... merge conflicts in the two files mentioned ... @unutbu just rebase and good 2 go

thank you

No, thank you. That was fun!
TST: added null conversions for TimeDeltaBlock in core/internals related (GH4396)
State: closed

closes #4396

NaT wraparound bug again, fixed


Labels:

Comments:
cc @yarikoptic

I am pretty sure this fixes the open sparc issues. Except for 1 thing.

We are doing a test for clipboard access, which uses xclip. But since there isn't a display attached this (and the plotting tests) fail. I am not sure that we can catch this. 

we test using a virtual frame buffer, can you test doing something like (need to install X)

```
export DISPLAY=:99.0
/etc/init.d/xvfb start
```
BUG/CLN: numpy compat with pandas numeric functions and cln of same (GH4435)
State: closed

related #4787 (keywords now pass thru to _reduce), default for `numeric` is still `None`

closes #4435

BUG: provide numpy compatibility with 1.7 when implicity using `__array__`, IOW:

`np.prod(s)`, or `np.mean(df,axis=1)` will now work (because numpy _decided_ to try to see if the
object has this function and then pass all kinds of keywords to it, so pandas gets the right function called
with `dtype`, and `out` passed in (currently ignored)

CLN: refactored all numeric and stat like function (sum/mean/mad/min/max) and cums (cummin/sum..)

to `core/generic.py` from Series/DataFrame/Panel


Labels:

Comments:
first PR with big deleted loc!

@jtratner don't worry....going to wait for your big ops merger first! (though I don't think much will conflic)

@jreback I know, nice to finally see something decrease the size of the codebase. Arithmetic refactor ended up about even if you exclude tests and docs changes.
CLN/ENH: Provide full suite of arithmetic (and flex) methods to all NDFrame objects.
State: closed

There's a lot of overlap right now, this is the first step to trying to
make this cleaner.
- Abstract all arithmetic methods into core/ops
- Add full range of flex arithmetic methods to all NDFrame/ndarray
  PandasObjects (except for SparsePanel pow and mod, which only work for
  scalars)
- Normalize arithmetic methods signature (see
  `ops.add_special_arithmetic_methods` and
  `ops.add_flex_arithmetic_methods` for signature).
- Opt-in more arithmetic operations with numexpr (except for
  SparsePanel, which has to opt-out because it doesn't respond to
  `shape`).
- BUG: Fix `_fill_zeros` call to work even if TypeError (previously
  was inconsistent).
- Add bind method to core/common

Closes #3765.
Closes #4334.
Closes #4051. 
Closes #5033.
Closes #4331.


Labels:

Comments:
This could use more work later on, but this is a nightmare to maintain in the interim, so I'd like to put it into pandas. Comments?

To be clear, basically the only changes in functionality in this PR are:
- Series and Panel use numexpr sometimes
- change to the signature for arithmetic methods
- add a few more flex methods.

@cpcloud @jreback any last fixes? Again, these are relatively minimal changes, aside from moving functions.

ok by me.

ok here too
ENH: PySide support for qtpandas.
State: closed

Added PySide support for the qtpandas DataFrameWidget.


Labels:

Comments:
Please add a note to doc/source/release.rst (which you can just squash into one commit) and it'd be great if you could change the name of the commit to `ENH: Add PySide support for qtpandas.` 

I would move this out of sandbox maybe to pandas.utils?

@jreback In my opinion it's still just an example... Lacks a lot of features to be a full DataFrame widget... (adding/deleting rows and columns, better type support, better edit, etc...)

oooh haha didn't realize this was sandbox. Your call if you want to put in release notes.

@jreback Done.

Can we stick a little doc section somewhere that describes this ? Maybe visualiztion? @cpcloud  where do you think

`visualization.rst` is fine

Should I open a new issue? "DOC: Add docs for qtpandas"?
I don't know what should I write there? :) This was just a support thing for PySide. :)

Maybe I can write a paragraph like: If you want to visualise your DataFrame in a Qt application you can use the DataFrameWidget like this: ... example ... And thats it :) But I would need to create a new section: Visualizing data in Qt... Or something like that...

@alefnula up to you whether u want to create a new issue. what you just suggested sounds fine

@alefnula just create a small section in visualization, you can include in this PR

Added.

@cpcloud @jtratner  ?

ok with me

looks like it needs one more rebase then good 2 go

@alefnula thank you
BUG: wrong index name during read_csv if using usecols
State: closed

Closes #4201

If user passes usecols and not names, then ensure that the
inferred names refer to the used columns, not the document's
columns.


Labels:

Comments:
@guyrt thanks again!

another issue for you! https://github.com/pydata/pandas/issues/4363

and one more! https://github.com/pydata/pandas/issues/4880

@guyrt any luck with either of these?

haven't looked at them yet - working on 4335
added halflife to exponentially weighted moving functions
State: closed

Currently for the exponentially weighted moving functions (ewma, ewmstd, ewmvol, ewmvar, ewmcov) there are two ways (span, center of mass) to specify how fast the exponential decay is. It would be nice to support a "half life" option as well.

The half life is basically just the number of periods in which the exponential weight drops to one half, i.e.,

(1 - \alpha)^h = 0.5, h: half life


Labels:

Comments:
can you add an entry in `doc/source/release.rst` (reference this PR, but use the issue tag), enhancements section. thanks

I've added an entry to the release.rst for this, thanks! 

gr8. can you squash down to a single commit?
https://github.com/pydata/pandas/wiki/Using-Git

Sure it's squashed down to a single commit now

thanks!
BUG: Warn when dtypes differ in between chunks in csv parser
State: closed

Closes #3866


Labels:

Comments:
I've pushed what I think is a good compromise.

If all types are numeric, then coerce appropriately.

If not, then use np.object type (this is actually the default) and issue warning. The problem here is that an array with type object can hold elements of different types. We either have to coerce the types to the same (very complicated with dates, objects, and what have you) or issue a warning and let the user sort it out. I've opted for a warning in this case. Tests added for both cases.

ok....that sounds good. coercing is quite difficult, I fixed a bug in concat that does that and its about a page of code!. The parser is not setup to do it (nor should it be), so that is fine.

The warning is probably fine. Though I don't believe other warnings are issued for really anything. Maybe they should even occur in other situations like this. If you think of something, pls open an issue.

thanks!

any chance that code in concat can be used for this?

maybe
look in tools/merge/concat_single_item

not sure it is worth it though

Done. Should be ready to go pending tests.

I made two changes:
Fix extra print (oops)
Change tests to run on all parsers. Only C low memory parser should raise the warning.

gr8.....waiting to merge a big change (unrealed to this)....

@guyrt #4335?

I'll take a look at that one next.

I've got a data set I've been playing with (nhtsa vehicle safety reports... pretty interesting) that triggers the warning for 11 columns. That's 11 warnings. I'm going to try to squash it to one warning, so hold off on merging this request until I do.

ok...sure...lmk

Tests pass. Ready for merge.

thank you sir!

@guyrt did you have a chance to look at #4335 ?

@jreback  yes - I'll add comments there.
CLN: fix py2to3 issues in categorical.py
State: closed


Labels:

Comments:
how is it possible that these passed import? They should fail in Python 3.2 as syntax errors!

they are! but I think that original PR must have been before your 2-to-3 changes so it look 'passed'

when I merged them in it passed in 2.7 ok (as the u"") syntax is ok in 2.7
BUG: Bug in concatenation with duplicate columns across dtypes, GH4975
State: closed

partially fixed in #4771

closes #4975

```
In [1]: w = DataFrame(np.random.randn(4,2), columns=["x", "y"])

In [2]: x = DataFrame(np.random.randn(4,2), columns=["x", "y"])

In [3]: y = DataFrame(np.random.randn(4,2), columns=["x", "y"])

In [4]: z = DataFrame(np.random.randn(4,2), columns=["x", "y"])

In [5]: dta = x.merge(y, left_index=True, right_index=True).merge(z, left_index=True, right_index=True, how="outer")

In [6]: dta = dta.merge(w, left_index=True, right_index=True)

In [7]: dta
Out[7]: 
        x_x       y_x       x_y       y_y       x_x       y_x       x_y       y_y
0  0.393625 -0.340291  0.035043 -0.195235 -0.892856 -0.357269  0.820424  0.142803
1 -1.600176  0.737261 -0.571140  1.352393  0.201634  1.403633  0.590919 -1.003057
2 -1.046113  2.148139  2.406527 -1.460300  0.881712  0.949246 -0.061758 -0.386265
3 -0.472761 -0.055612 -0.449152 -0.209876  1.076689  0.294275 -0.684433  0.925683
```

Via direct concat

```
In [11]: concat([x,y,z,w],axis=1)
Out[11]: 
          x         y         x         y         x         y         x         y
0  0.393625 -0.340291  0.035043 -0.195235 -0.892856 -0.357269  0.820424  0.142803
1 -1.600176  0.737261 -0.571140  1.352393  0.201634  1.403633  0.590919 -1.003057
2 -1.046113  2.148139  2.406527 -1.460300  0.881712  0.949246 -0.061758 -0.386265
3 -0.472761 -0.055612 -0.449152 -0.209876  1.076689  0.294275 -0.684433  0.925683

In [8]: expected = concat([x,y,z,w],axis=1)

In [9]: expected.columns=['x_x','y_x','x_y','y_y','x_x','y_x','x_y','y_y']

In [10]: expected
Out[10]: 
        x_x       y_x       x_y       y_y       x_x       y_x       x_y       y_y
0  0.393625 -0.340291  0.035043 -0.195235 -0.892856 -0.357269  0.820424  0.142803
1 -1.600176  0.737261 -0.571140  1.352393  0.201634  1.403633  0.590919 -1.003057
2 -1.046113  2.148139  2.406527 -1.460300  0.881712  0.949246 -0.061758 -0.386265
3 -0.472761 -0.055612 -0.449152 -0.209876  1.076689  0.294275 -0.684433  0.925683

```


Labels:

Comments:
@jseabold the issue you report in #4975 was an actual bug (that was 'partially' fixed in #4771)

thanks!
BUG: allow Timestamp comparisons on the left
State: closed

closes #4982 


Labels:

Comments:
whoosh you went all in on this! fused types!

yeah ... there's an annoying vbench hit tho with index joins ... trying to find the source of it ... i actually got rid of the fused types, not necessary since really just `int`, `datetime` and `Series` are th possibilities

ahh...ok

@jreback comments?

otherwise looks ok

cool, i'll merge after next travis pass...
TST/CLN: pare down the eval test suite
State: closed

Current running time of the `eval` test suite (not including `query`) 

```
$ nosetests pandas/computation/tests/test_eval.py                                                   1 ↵
......../home/phillip/Documents/code/py/pandas/pandas/core/frame.py:3088: FutureWarning: TimeSeries broadcasting along DataFrame index by default is deprecated. Please use DataFrame.<op> to explicitly broadcast arithmetic operations along the index
  FutureWarning)
.........................................................................................S.......................................................................................................................
----------------------------------------------------------------------
Ran 217 tests in 190.685s

OK (SKIP=1)
```

New running time :speedboat: :

```
$ nosetests pandas/computation/tests/test_eval.py
......../home/phillip/Documents/code/py/pandas/pandas/core/frame.py:3088: FutureWarning: TimeSeries broadcasting along DataFrame index by default is deprecated. Please use DataFrame.<op> to explicitly broadcast arithmetic operations along the index
  FutureWarning)
.........................................................................................S.......................................................................................................................
----------------------------------------------------------------------
Ran 217 tests in 39.299s

OK (SKIP=1)
```


Labels:

Comments:
can u include alignment tests at least on lhs for all index types
or does it make it take much longer?

let me see

okay doesn't add that much (about 3s) to do lhs all index types in the alignment tests (sans `PeriodIndex`)

also unmarked a few `@slow` tests so that the basic alignment tests would run across all our builds except the slow which will run the more complicated tests

gr8 I don't think we have the index interactions anywhere else so that's good

@jreback merge?

yep
CLN: change print to print() in docs
State: closed

closes #4967

`print` statements refactoring


Labels:

Comments:
@cpcloud can you build these docs just to check?

@alefnula fyi....in fut make the top of the PR like I did here; then when merged willl auto close the issue
and make the title reflect the issue for easy viewing

thanks! these r gr8

@jreback sure thing

hm `numpydoc` not compatible with py3 :(

at least the version we're using isn't

@cpcloud But this is also valid py2 code :-/

yep those are building

@jreback there's that `to_datetime` failure (@alefnula, unrelated to ur changes) but otherwise things are :ok_hand: in py2 land

right, I think dan birken still fixing that up
API: raise a TypeError on invalid comparison ops on Series (e.g. integer/datetime) GH4968
State: closed

closes #4968


Labels:

Comments:
Just as an FYI as to why this PR is working:
1. We have `__eq__` using `operator.eq` which will return the result of calling an object's overridden `tp_richcompare` method _or_ a `bool` (if the former lhs and rhs methods return `NotImplemented`). This is because `operator.eq` compares pointers if both the left-hand-side and right-hand-side methods return `NotImplemented`. In this case, `ndarray.__eq__` is return `NotImplemented` for both the left _and_ right hand side. Python then proceeds to compare object pointers.
2. The `wrapper` function receives a `bool` from `na_op`. `wrapper` constructs the result with the left-hand-side's index which will fill the result with a single value if it's passed

@jtratner @cpcloud ok to merge then....?

hold on 1 sec want to try something

yep okay, good 2 go
ENH Change ExcelFile to accept a workbook for the path_or_buf argument.
State: closed

That will also mean that read_excel accepts workbooks too. (closes #4961 and #4959)


Labels:

Comments:
It might make sense to add an `engine` argument to both the `ExcelFile` and `read_excel` constructor that has to be passed along with a book...not sure.

@jtratner It seems to make sense to add the engine argument, et. al. in another PR. Thoughts?

@cancan101 I was thinking that we would require passing an engine argument with a book, so that, if we were to add other engines going forward, we wouldn't have to try isinstance checks for determining which backend to use.

So in other words you have `engine=None`, an if statement that checks for whatever conditions were previously checked for path_or_buf and then a final else clause that tells you that you have to specify an engine if not passing a buffer or path.  Not totally sure on that, but that's what I was thinking.

@jtratner @cpcloud I pushed a change to test `read_excel` and renamed `path_or_buf`. I am still a little unclear on how to deal with an `engine` param for now. For the time being I can map both None and "xlrd" to go through the current code path and anything else to raise and error. Not sure how useful that is though.

@cancan101 As different excel reading backends might have slightly different APIs for excel workbooks, then passing an `engine` should really be a requirement if you want to pass a workbook in as `io`. More clear? For example if I have an excel workbook that I've created using `openpyxl` (an instance of `Workbook`), that's different from the `Book` class used by `xlrd`. One would need a way to differentiate between those, and if I understand @jtratner then that's exactly what engine would do.

yep, that's exactly what I mean (even though for right now it's all xlrd). Thanks for laying that out!

@jtratner @cpcloud Okay. I pushed something for `engine` support. Let me know if this isn't right.

I'll take a look soon.

On Wed, Sep 25, 2013 at 7:33 PM, Alex Rothberg notifications@github.comwrote:

> @jtratner https://github.com/jtratner @cpcloudhttps://github.com/cpcloudOkay. I pushed something for
> engine support. Let me know if this isn't right.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4962#issuecomment-25133440
> .

ping!

@cancan101 thanks for the ping. Apparently you need to rebase this too. (so sayeth github)

@jtratner Pushed.

after this passes Travis I'll merge.
WIP: Refactoring SQL code into per-flavor classes.
State: closed

Still don't have down the code to dispatch appropriately (though I'm not
even clear why the existing code has individual flavors, given that it
appears to only accept SQLAlchemy objects anyways). Also would be
helpful to cover additional cases where we get connections and want to
convert to cursors.


Labels:

Comments:
@jtratner , superceded by #5950?

Yes
BUG: Fixed _format_labels in tile.py for np.inf.
State: closed

Fixes issue https://github.com/pydata/pandas/issues/3415.


Labels:

Comments:
can you squash to 1 commit? looks good 

if you want to put you test example in v0.13.0 you can (to sort of 'announce' this fix)..up2u

I did that, I hope adding the example to the documentation is okay.

One question: It's rather unmathematical to write `(0, inf]` (i.e. closed at infinity). Should we be correct about this and make the label `(0, inf)`, resp. `(-inf, 0)`, or do we prefer the uniformity?

@jtratner @cpcloud ?

@filmor Can u rebase?

Done. The question still stands though, but it can be handled in a different PR.

I think it's more consistent to use the existing labeling scheme (even if technically mathematically incorrect)

That's fine with me :)

@filmor thanks!
BUG: pd.isnull with tuple argumnet should return same results as list (#4872)
State: closed

Fix for issue #4872. Now tuple argument for pd.isnull behaves the same way as list pd.isnull(list(tuple_argument)).


Labels:

Comments:
@ssalonen just so you're aware, I'm not sure whether we would accept this PR, because it introduces inconsistencies between how tuples are treated when part of an index or other NDFrame objects vs. the toplevel. Generally, in pandas lists indicate `you should iterate over me` whereas tuples indicate `you should consider me to be one scalar element`.

Further, this would mean that this will no longer be true (and if it's not currently, it should):

``` python
ind = Index([(1, 2), 5, (np.nan,)])
assert all(a == b for a, b in zip(map(pd.isnull, ind), ind.isnull())
```

that said, we'd definitely incorporate the test cases (altered to reflect current behavior), because it's useful to establish how tuples are supposed to be treated.

@jtratner after thinking about this for a while I think I agree with you about how tuples are handled. 

My base claim was that the behaviour should be the same as in numpy but I think in numpy having tuples in a array are more of a corner-case and not supported properly (try for example creating array with tuple elements in numpy and pandas...).

When accepting the fact that Series and Dataframes with tuple elements are OK and part of the framework, isnull should handle tuple as scalar. This is because we would like to have map(series, pd.isnull) to equal pd.isnull(series)

I do believe that a disclaimer in the docs might be in order...

@ssalonen ok why don't u close this and I will change original to a doc update (maybe in missing.rst)

closing and changing to a doc issue
BUG: Conflict between thousands sep and date parser.
State: closed

closes #4678 
closes #4322 

I've fixed the C parser portion. The issue there was that it did not handle the case where parse_dates is a dict.

Python parser fix yet to come. That test still fails.

Example:
    s = '06-02-2013;13:00;1-000,215;0,215;0,185;0,205;0,00'
    d = pd.read_csv(StringIO(s), header=None, parse_dates={'Date': [0, 1]}, thousands='-', sep=';', decimal=',')

Then d should have a column 'Date' with value "2013-06-02 13:00:00"


Labels:

Comments:
@guyrt go ahead and put in this same PR.....(both c and python)

@jreback I've pushed a fix for Python parser, but I'm not completely happy with it.

The issue is that the Python parser has special handling for the first line when headers=None (https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L1414). In this section, we read the first line without having access to the column names in every case. Therefore, if a column is identified by name as in a date (so we skip thousands parsing) then we are unable to identify which column to skip. This is a pretty rare case requiring:
- no header
- parse_dates = a dict that refers to columns by name
- those columns use the thousands separator.

I don't see an easy way to fix this problem without rethinking the first line processing completely.

hmm....but in that case you would HAVE to have `names` specified (otherwise its an error to put the name in parse_dates)...pretty sure that is available in the passed kwds

You would have to now. Previously, any use of self.names was deferred until after the first line was read so names could be inferred set. Now, we have to be prepared to read that line and selectively remove thousands separators, so we have to use names before the first line is read.

makes sense...whole ordering bizness is somewhat complicated....mk if you need help

Turns out there was an easy fix. The first line is pushed into a buffer, but it isn't fully processed until later. I deferred checking for thousands until later. 

gr8
ping when u think ready

@jreback tests pass

do both test cases from the issue pass? or is there something else going on?

There were two open issues:
#4322 (thousands separator)
#4678 (date/thousands conflict)

Both are fixed. 

does this also fix #4382 ?

So such luck on #4382 

can you add the #4322 reference to the release notes and a test for that as well?

#4322 is already in the release notes and was tested in a previous PR.

What happened is that I fixed 4322 (thousands operator ignored) which uncovered this bug about a conflict between dates and the thousands operator. Someone reopened 4322 but also created a new ticket, so we can close them both.

@guyrt thanks!

@guyrt interested in #4335, #4201?

I'll take a look. Working on #3866 right now