BrandonFors/pandas-github-issues huggingface dataset:
BUG: fillna with method segfaults on zero-length input (fixes #2775)
State: closed

fixes #2775

just added a check for zero-length data to the backfill and pad templates

not sure if I should add test coverage? the problem is that the tests will not fail without the fix, but rather segfault, so it might not be a good idea


Labels:

Comments:
fyi....this is going to conflict with dtypes branch....maybe do this afterwards? #2708

do you want to just fix it on your branch instead and we can close this PR? it's just some checks for N == 0

sure...just pad & backfill (all methods)....do we have a test to replicate?

just run `fillna` on any `DataFrame` or `Series` with zero rows, using a method rather than a fill value...
example from original issue was `pandas.DataFrame(columns=["x"]).x.fillna(method="pad", inplace=1)`

I don't think inplace is needed to trigger, and I was unable to replicate with Series().fillna(method="pad", inplace=1)

Not sure why pandas.DataFrame(columns=["x"]).x and series() behave differently, there may be a deeper issue
at play.

it triggered for me with and w/o inplace....adding as a test

ok...fixed in #2708

fyi, apparently it only segfaults when `dtype=='object'`, probably because that's the only case in which the value is dereferenced as a pointer. for non-object dtypes, the bug is still there, but it doesn't cause a segfault because the memory is just being read and interpreted as an integer/float

``` python
In [2]: p.Series().dtype
Out[2]: dtype('float64')

In [3]: p.Series().fillna(method='pad')
Out[3]: []

In [4]: p.DataFrame(columns=['x'])['x'].dtype
Out[4]: dtype('object')

In [5]: p.DataFrame(columns=['x'])['x'].fillna(method='pad')
Segmentation fault (core dumped)
```

``` python
In [2]: p.Series().astype('int64').fillna(method='pad')
Out[2]: []

In [3]: p.Series().astype('object').fillna(method='pad')
Segmentation fault (core dumped)
```

Should this be patched in a v0.10.2 release? I think that might happen before #2708 is merged into v0.11 dev branch

up 2 you....it actually hard to trigger this (though fix is pretty trivial too)...

@wesm, i restored the branch and commited the same test as in #2708, in case you want to merge this first. there will probably be conflicts later if you do, but they'll be easy to resolve

closed because of merge of #2708
BUG: Mismatch between get and set behavior for slices of floating indices (fixes #2727)
State: closed

fixes #2727: `_NDFrameIndexer._convert_to_indexer` modified to match behavior in `_NDFrameIndexer._get_slice_axis`, new tests added to existing unit test


Labels:

Comments:
Was informed that I could have forced the rebased push through rather than making a new branch, will do that next time

cherry-picked. thanks
BUG: Various inconsistencies in DataFrame getitem/setitem behavior (fixes #2765)
State: closed

fixes #2765 

Note, to address `__getitem__` does not reindex boolean Series key but `__setitem__` does ", I have changed the `__getitem__` behavior rather than the `__setitem__` behavior because it is more consistent with other indexing behavior and does not break any tests...just in case, I have inserted a warning about this happening (which only triggers when a reindex is actually required)

Also, in addition to resolving the bugs, I have refactored the code to make `__getitem__` and `__setitem__` code paths as symmetric as possible, in order to help prevent more issues in the future.

Unit test coverage added, and performance tests show no impact


Labels:

Comments:
(Will avoid the new PR next time...didn't realize I could force the rebased push without breaking GitHub)

cherry picked this. thanks
BUG: Various inconsistencies in DataFrame getitem/setitem behavior (fixes #2765)
State: closed

see #2765 

Note, to address " ___getitem___ does not reindex boolean Series key but ___setitem___ does ", I have changed the ___getitem___ behavior rather than the ___setitem___ behavior because it is more consistent with other indexing behavior and does not break any tests...just in case, I have inserted a warning about this happening (which only triggers when a reindex is actually required)

Also, in addition to resolving the bugs, I have refactored the code to make ___getitem___ and ___setitem___ code paths as symmetric as possible, in order to help prevent more issues in the future.


Labels:

Comments:
Last time I tumbled down the spaghetti stack of `__getitem__` I encountered a helpful 
"here be performance dragons" warning comment left somewhere by wesm for the potential 
yet-to-be scarred contributor.

So, just a hunch, but did you check for perf regressions for this vs. master? 
(using `test_perf.sh` is probably most convenient).

ok, will take a look

ok, only one test consistently affected: after a tweak, things are good now (test_perf.sh gives ratios that look basically normally distributed around 1.0, with no single test consistently doing worse across multiple runs)

ok, good.

If you're changing behaviour of `__getitem__` and `__setitem__`, it's a good idea
to reify the desired behaviour with some new tests, since there's obviously 
no coverage there right now and It'll make it easier to review as well.

yeah, already added test coverage

Sorry, I missed that when looking.

found a leftover print statement in my test code...decided to consolidated the merges and made a new PR at #2776 rather than spamming the commit log

Github allows you to do a forced push to the PR branch, in which case, the current state of the branch
replaces the original PR commits. No need for a new PR (next time).

Ahh, wasn't sure if Github handled non-ff pushes to PR branches well. What does the PR show on the website when you do that? I guess I'll find out next time.

It throws away the old commits and shows just the new state of the branch.

ok, good to know
BUG: DataFrame.clip arguments inconsistent  (close #GH 2747)
State: closed

arguments are inconsistent to numpy and Series.clip


Labels:

Comments:
This obviously breaks back compat.
Maybe cheat and just use `min(upper,lower)` and `max(upper,lower)`? 
That way both the old code and the numpy convention will work.

I think this SHOULD break compat, as it very self-inconsistent....how about a warning with the max/min as the test? 

Change the args around, but use min/max anyway. fix it _and_ keep old code working.

fixed

Changed to 0.10.2,  now that it's not a breaking change.

Thank you guys

Merged. thanks
A patch to make pandas install and use numpy if it doesn't find it on the target host
State: closed

Pandas will normally fail to install if numpy is not installed on the target host. This should normally be resolved by specifying numpy as a dependency but Pandas will still require that numpy is installed first due to its need of the numpy include files.

This patch resolves this problem by changing the way the numpy include path is retrieved so that it is not necessary that numpy is first installed before pandas - they can both be installed together; numpy coming first of course.


Labels:

Comments:
thanks! cherry picked this one
ENH: add option chop_threshold to control display of small numerical values as zero
State: closed

Doesn't cover complex numbers for now (no formatter implemented in format.py)

I think the "supress" kwd choice in numpy is not that awesome. So I borrowed `chop` from 
the mathematica parlance.
#2734


Labels:

Comments:
It does what I asked - Thanks!

There is one improvement I think we could make. Assuming I have set the `precision` option to 5, when displaying a chopped entry near other data in scientific format (something like `2.54443e-03`), the chopped value will display as`0.00000e+00`. It would be nice if it just printed `0` instead of `'0.' + '0' * precision + 'e+00'`.

A side note, is there a way to control if things are printed in scientific format (with the e's)?

You can look at the code for `pandas.set_eng_float_format` to see how it's done,
and you can pass your own formatters to `to_string()` for even more control.

cherry picked. thanks

added mention to v0.11.0.txt and pushed.
ENH: provide squeeze method for removing 1-len dimensions, close (GH #2544)
State: closed

```
In [1]: p = Panel(randn(3,4,4),items=['ItemA','ItemB','ItemC'],
   ...:                        major_axis=date_range('20010102',periods=4),
   ...:                        minor_axis=['A','B','C','D'])
   ...:

In [2]: p
Out[2]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 4 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2001-01-02 00:00:00 to 2001-01-05 00:00:00
Minor_axis axis: A to D

In [3]: p.reindex(items=['ItemA']).squeeze()
Out[3]: 
                   A         B         C         D
2001-01-02 -1.182806 -1.835015 -0.008415  1.665945
2001-01-03 -0.347332 -0.257820  0.167772 -1.193518
2001-01-04 -0.502643  0.055733  0.733034 -1.070536
2001-01-05 -1.900588 -1.728706  1.060564 -1.138837

In [4]: p.reindex(items=['ItemA'],minor=['B']).squeeze()
Out[4]: 
2001-01-02   -1.835015
2001-01-03   -0.257820
2001-01-04    0.055733
2001-01-05   -1.728706
Freq: D, Name: B
```


Labels:

Comments:
cherry picked. thanks
value_counts() can now compute relative frequencies.
State: closed


Labels:

Comments:
Can you please add a test case?

Test case now provided ;)

Deferring til next release. Need to think about whether `relative` or `normalize` is the right argument name (should consult other sources for guidance)

I think `normalize` is more appropriate here. An individual frequency may be relative (hence the phrase "Relative frequency histogram"), but the overall is normalized. [citation needed]

Here's what matplotlib & numpy do:

http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.hist
has a "normed" argument.

http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html
has a "normed" argument.
apparently this is being deprecated for numpy 2.0 and will be using the "density" argument instead.

Both of the above links are worth reading for further clarification of how they handle this.

It seems a bit weird to talk about "density" with the discrete-sounding value_ **counts**, but I can see how it could make other things consistent... density would certainly be a good addition when bucketing.

_`normed` is better than `normalize` (and has the same meaning)_

Barring loud outcry, I plan to merge this using "normalize" as the keyword very soon.

+1

Thanks @lexual, merged as  b143a21. Apologies for the [delay](http://martinfowler.com/bliki/TwoHardThings.html).

Saw your django repo - good feature. Suggest you see if there's common ground with #2717.

Thanks @y-p . 

Do you have an opinion on whether something like https://github.com/lexual/pandas-love-ponies should live outside Pandas or inside Pandas? I never got any responses to my post to the mailing list a month ago.

My opinion is that pandas is in constant feature-creep japordy, and this
is not close to home. However, playing nice with SQL, especially under
a unified interface through SQL alchemy would be a win, so if django
fits in there as a "flavor", :thumbsup: , else, probably not core material.

others may disagree.

My code is completely Django specific so imagine it would be separate to sql alchemy work (although I haven't had a close look at the sql alchemy stuff). So I imagine you're probably right that it should live outside of Pandas unless it starts getting widespread use.
Added metadataframe.py which has MetaDataframe class.  This class stores...
State: closed

... a dataframe and attempts to overload all operators and promote dataframe methods to object methods in hopes that users can subclass this effectively as if it were a Dataframe itself.


Labels:

Comments:
Hi guys.  I'm not exactly sure what tests to run on this class.  Any help would be appreciated.  

One thing I'm worried about is that I didn't overload every attributes that DataFrame relies on.  I did most of the basic ones, and all those of the public API, but there may be more that I missed because they were inherited from NDFrame or PandasObject.

Also, I decided to to call my deserialization functions mload and mloads to avoid name conflicts with the pandas API.

Why not MetaDataFrame instead of MetaDataframe, to be consistent with DataFrame capitalization?

Not sure why I did this initially but will change it thanks for the heads
up.
On Jan 28, 2013 3:30 PM, "stephenwlin" notifications@github.com wrote:

> Why not MetaDataFrame instead of MetaDataframe, to be consistent with
> DataFrame capitalization?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/2695#issuecomment-12802717.

Lots of discussion (#2845) but no real concensus or champion. Users
are free to reuse this independently of it being merged into pandas.

If there are no objections, closing in 3..2..

I will keep the metadataframe module hosted separately in my PyUvVis (https://github.com/hugadams/pyuvvis) package and continue to use it and develop it with the intention of general use cases.  I monitor the mailing list frequently so any requests for it can be directed to me:

hughesadam87@gmail.com

When I get around to writing the PyUvVis official documentation, I'll put a small writeup together on this class by itself and share.  If there ever becomes a point that merging would be desired, that is fine.

Thank you. closing now.
BUG: series assignment with a boolean indexer
State: closed

- fixed assignment with a boolean indexer and a non-scalar as the value (see below)
- 2nd commit fixed the alignment issue; that is if the value is not a full series, align first (using where machinery)
- add neg and inv methods
- _check_bool_indexer now coerces to bool if needed (otherwise weird effects when doing neg/inv on boolean series)

```
In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: x = pd.Series(index=np.arange(10))

In [4]: y = pd.Series(np.random.rand(10),index=np.arange(10))

In [5]: y
Out[5]: 
0    0.197830
1    0.134425
2    0.766618
3    0.057765
4    0.790927
5    0.153636
6    0.585305
7    0.906787
8    0.647155
9    0.494154

In [6]: x[y>0.5] = y

In [7]: x
Out[7]: 
0         NaN
1         NaN
2    0.197830
3         NaN
4    0.134425
5         NaN
6    0.766618
7    0.057765
8    0.790927
9         NaN
```


Labels:

Comments:
thanks jeff
BUG: HDFStore fixes
State: closed

- shape attribute on GenericStorer returns tuple correctly now (not string)
- data_orientation was potentitally returning non-ints in a tuple (fixed downstream), fixes GH #2671, though fixed downstream anyhow
- fix for contains with a non-natural name (GH #2694 )
- added ability to read generic PyTables flavor tables
- added PerformanceWarning to trigger on putting of non-endemic types (only on Storers)
- correctly store float32 dtypes (when not-mixed with float64, otherwise converted)
- can close  GH  #1985 as well


Labels:

Comments:
thanks jeff much appreciated as always

Thanks jreback, float32s were on my wishlist :)

I would like to get started contributing especially in
the pytables integration area. Is there a roadmap or some type
of docs other than source for pandas.io.pytables?

great 2 hear!

issue #2391 is an older list of todos

a couple of possibilities:

real dtype integration is coming on PR #2708 (eg even though 0.10.1 will actually read/write float32 columns u can't really do much with them w/o having them upcasted) - in any event I think HDFStore will accommodate this already

one big area is to test whether data columns really are slower; it thus may make sense to make data columns = True the default (but not necessarily index them)

are u familiar with R? I think could make export a bit easier (there is a section in docs, but maybe a dedicated option. / function)

have some more thoughts but maybe will start a new issue

I updated #2391 with some ideas....feel free to suggest based on your use cases...
add support for microsecond periods (GH2145)
State: closed

(Continues #2145)

Alrighty. This should be fine. There may be a few formatting changes that I rolled back through the git diff / apply patch, but all not-skipped tests pass so I figured it wasn't a big deal. Here's what I did to fix the problem, which was continually merging with upstream:

``` shell
function get_my_commits {
    local curdir="$(pwd)"
    cd "$1"
    # %n adds a newline in addition to the file name
    # sed removes blank lines here
    # sort -u sorts and removes duplicates lines
    git log --name-only --pretty=%n --author=Phillip 0eeff..master | sort -u | sed '/^$/d' 
}

old_pandas=/path/to/evil/pandas/merge/of/death
new_pandas=/shiny/new/checkout/of/pandas

get_my_commits $old_pandas | while read file; do
    # some boolean tests to filter out setup.py and missing files between versions
    diff -pu "$old_pandas/$file" "$new_pandas/$file" | patch -p0 "$new_pandas/$file"
done
```

Needless to say I won't be doing any more merging of upstream into my branch. Thanks to those who helped me out here.


Labels:

Comments:
@cpcloud, I saw an emacs related commit in the original pile. 
If you're an emacs user, you might like to look into [magit](https://github.com/magit/magit). 
It has fantastic interactive rebase support and I use it all the time.

@y-p Cool, yeah I saw your comment before. Thanks! I've used magit a little bit (comes with emacs-starter-kit, I'm definitely a huge emacs fan), but I'm sort of a command line junkie and I haven't spent the time to get used to magit yet. After this experience, however, I will start using it.

Quick question: if I `git pull --rebase upstream master` on this branch will that add more commits to this pull request, i.e., will it be problematic?

If you just add commits to the branch and push, they'll be added to the PR.
If you do a rebase,you'll need to do "git push -f" since you'll be clobbering the 
existing commits. Forced push is usually a no-no but usually fine for pull request branches.

Make sure you know how to backpedal if the rebase has conflicts and you make a mistake,
or use a temporary branch.

Is this going to make it into 0.11?

no, and I think #3060 swallowed it.

@y-p is correct. I think @wuan has a more general solution to this issue. Should I retract the pr associated with this issue? 

I'm going to close and look at merging #3060
add support for microsecond periods
State: closed


Labels:

Comments:
Hurf. I wonder if this mess can be salvaged by cherry-picking all of your commits ("manual rebase") onto master?

Yeah, sorry about that. I should have asked before merging with upstream. How would I go about this? I thought I was past the git n00b stage, but I guess not.

If you have to, rebase on top of upstream. Do _not_ merge

Can you try to convert the diff into a single commit on top of master? I don't care that much about preserving the code history but would rather not have the merge commits

So something like `git diff master 0eeff49` and then make a patch out of that, fork the current pandas git master and apply the patch then resubmit?

Yes

Sent from my mobile device

On Jan 8, 2013, at 6:22 PM, Phillip Cloud notifications@github.com wrote:

So something like git diff master 0eeff49 and then make a patch out of
that, fork the current pandas git master and apply the patch then resubmit?

—
Reply to this email directly or view it on
GitHubhttps://github.com/pydata/pandas/pull/2662#issuecomment-12023015.
Writing doc about I/O SQL features. See #2541
State: closed

Hi there, 

I wrote some contents about I/O SQL pandas features:
- some examples about the `read_frame` function with SQLite ;
- list other functions such as `tquery` (without example) ;
- write a tiny note about `write_frame` function.

I'm not sure about the title of this section. Any proposal is welcome ! Don't hesitate to comment the content (typo, misunderstanding, ...)
## Cheers

Damien G.


Labels:

Comments:
closes #2541 

thanks!
added a compression argument to to_csv to be sent to _get_handle
State: closed

Writing csv files with gzip compression is very useful as it both minimized the disk size taken by large data files, and can be easily read by _R_, often faster than the same csv file without compression. 
Reading compressed csv files in _R_ is done without any additional input by the user, and is also already supported in _Pandas_


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
Needs a test case

Can you suggest where a test case should be written?

pandas/tests/test_frame.py has all the `test_to_csv_*` tests.

I intended to write a roundtrip test, but `from_csv` doesn't support `compression`
despite it's hiding down the call stack. 
Should probably keep things consistent and update `from_csv` the same way.

OK. This is on my task list, but unfortunately not high enough...
Hopefully I will get to it soon.

MIA.

@y-p I'd like to reopen this and complete the PR.
I notice that `read_csv` quietly reads gzipped csv files, so if I understand correctly, I need to add a test for both `to_csv` using a roundtrip with `read_csv` in `pandas/tests/test_frame.py`.
Let me know if this is true and if this is still relevant.
Also - should I do anything because so much time passed, like rebase or just fork again and apply my minor changes, or would this be handled by you during merging?

you should rebase on master and open a new PR
quite a lot has changed since this issue came up

contributing docs are [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

and this issue is #7615
Try to fix BUG #2599
State: closed

Not sure if this is a good way to go about it, but it seems to work.

Might be good to get rid of all the INT64_MAX/MIN stuff in parser.pyx and replace with np.iinfo(np.int64).max/min instead.


Labels:

Comments:
I'm pretty sure this is a Cython bug also and should be reported separately

Agreed, i will file something with them.  However, I find it appealing to rely on numpy to provide the min and max values for the different dtypes rather than having to rely on stdint.h.   I think i should be able to eliminate the use of any of the stdint stuff from parser.pyx.  Would that be helpful or would you rather scrap the change altogether?

Merged and verified this fixes the issue. Thanks dieter
BUG: fixed old version compatibility warnings
State: closed

- added copy method to allow file upgrades to new version
- refactor to add Storer class to represent all non-tables
- add shape field to display for non-tables
- close GH #2637


Labels:

Comments:
sorry for the extra merges....i put a warning if you are reading an older version file....didn't have an easy way for users to upgrade (all you have to do is read in an out...easy..)..so added a method...

I put the data file in _static in the doc dir....however, not sure if you can make it a link to pandas/io/tests/legacy_0.10.h5 (its the same file)

no more changes - ready to merge......(and take my keyboard away) :)

@wesm should merge cleanly aftter #2632 (I had changed a lot of code, so most of the autopeping didn't matter i think)
ready to merge in any event

Merged, thanks!