BrandonFors/pandas-github-issues huggingface dataset:
GBQ: Updated Documentation, and added method to generic.py
State: closed

Added some documentation for GBQ module


Labels:

Comments:
i led u astray

move to_gbq to core/frame.py (from core/generic.py)
as you only allow data frames (no series or panels) 

@jacobschaer 

actually....instead of moving to `core/frame.py`, why don't you just check that the `obj.ndim==2` and otherwise `raise NotImlementedError`, that way you can easily support a series at some point in the future if you want, and you get a nice error message (and not `AttributeError`)

Not sure if that's the best choice. If we did choose to support it later,
introspection on the object in this version would suggest that this exists
and is supported.

@jacobschaer reverting back......so move `to_gbq` from `core/generic.py` to `core/frame.py`...makes the most sense (we have this with `to_stata/to_csv/to_sql` as they don't make sense generally for non-frames).

@jacobschaer can you fixup?

@jreback - Sorry about the delay. Docs should be fixed up, and the `to_gbq` method is in `core/frame.py`.

just a small change; pls squash and i'll merge this

@jreback - Fixed and squashed.

@jorisvandenbossche - Not entirely sure what needed to be done, but I think I did what you asked.

@jacobschaer Indeed, you did what I meant!
ENH: add to_frame method to Series
State: closed

Adding a method to Series to promote it to a DataFrame.  I've wished this existed so that when a method returns a series, and you could easily promote it to a DataFrame so that it's easier to add columns to it.  


Labels:

Comments:
pls also add a release notes entry (you can reference this PR for the issue number), enhancements section.

also add to the docs section in api.

and a 1-liner in v0.13.0.txt

pls change the name of the method to `to_frame`....for consistency with others ,e .g.`to_panel`....

See updated commit.  Thanks for all the feedback. 

@jreback let me know if the tests don't address the case you're concerned about

"also add to the docs section in api." how do I do this?

edit docs/source/api.rst

just like other files

on another note, how do I generate the docs for testing?

make doc
then view in a browser

http://pandas.pydata.org/developers.html

Updated commit should include docs update.

can u add same line in v0.13 to doc/source/release.rst as well

@cpcloud @jtratner comments?

just needs the name changed to `to_frame` then good 2 go

@drasch thanks!

doh, I just added the release.rst entry

no worries I added on the merge
DOC: Added filtering example to groupby doc
State: closed

closes https://github.com/pydata/pandas/issues/4630


Labels:

Comments:
can you rebase and squash these down?

see here: https://github.com/pydata/pandas/wiki/Using-Git

cherry-picked https://github.com/pydata/pandas/commit/eb49283409b04ee481dfa81a49d78c1c437b73f6

thanks!
PERF: remove auto-boxing on isnull/notnull
State: closed

- API/CLN: provide isnull/notnull on NDFrame objects
- PERF: remove auto-boxing on isnull/notnull

related #5135


Labels:

Comments:
hold on - this is a not-insignificant change here - doesn't this mean you lose indexing on isnull() and notnull()?   I think it's fine to have pd.isnull() return unboxed, but NDFrame.isnull() and NDFrame.notnull() should definitely box by default. Maybe it could just short-circuit the constructor instead?  The boxing behavior is available on 0.10 and later, so it's not reasonable to change without discussing. You lose a _lot_ of expressiveness this way. Maybe you could add a keyword argument "box=True" to it so that internals can call and not box result.

(I'm assuming it's available in 0.11 and 0.12, can't test it right now)

@jtratner I preserved the orginal behavior (I had actually changed this in internal refactoring for 0.13). the `isnull` called directly don't need to actually create a Series (I was going to add a `box` argument, but its not necessary.

This doesn't affect > 1dim objects anyhow didn't even have `df.isnull()`; `pd.isnull(df)` returns the correct (boxed version)
Here's 0.12

```
In [1]: s = Series([1,np.nan])

In [2]: df = DataFrame(dict(A = s))

In [3]: s.isnull()
Out[3]: 
0    False
1     True
dtype: bool

In [4]: df.isnull()
AttributeError

In [5]: pd.isnull(s)
Out[5]: 
0    False
1     True
dtype: bool

In [6]: pd.isnull(df)
Out[6]: 
       A
0  False
1   True
```

The difference is:
a Series will not box (this _happens_ to work in 0.12)

```
In [5]: pd.isnull(s)
Out[5]: array([False,  True], dtype=bool)
```

I could make this work (but will need a `box` keyword that is `True` by default by set `False` by almost
every routine that used this internally. Which I think is a needless complication.

The API is actually much cleaner as `isnull/notnull` exist on all objects.

and boolean indexing will work exactly the same. I don't actually think this is an API change at all.

hmm...on second thought it DOES box Series.....ok...easy enough to do
BUG: allow tuples in recursive call to replace
State: closed

This avoids the seeming passage of regular expressions

closes #5143.


Labels:

Comments:
also fixes a bug where a `Series` wasn't accepted because it has no `items` method

the beast that is `NDFrame.replace()` could use a makeover ... will do it 0.14

here's a similar bug, I think:

comes up in `test_replace_with_none_value`

shows up on sparc/2.6, because the dictionary ordering is different (not sure why)

```
In [1]: df = DataFrame(dict(A = [np.nan,1]))

In [2]: df.replace(to_replace={nan: 0, 1: -1e8})
Out[2]: 
           A
0          0
1 -100000000

In [3]: df.replace(to_replace=(1,nan),value=(-1e8,0))
Out[3]: 
           A
0 -100000000
1          0

```

I don' think you can do it with a single masking in `core/internals/Block/replace`. I think that you have to iteratore over the to_replace/values and replace one-by-one

it _does_ iterate one by one in `ObjectBlock.replace()`

ahh...but this is a `FloatBlock`, so doesn't!

oh fail ... ok ... it's becaue they are tuples

I was just simulating what your `keys, values = zip(*items)` does

right .... that's the bug

ahh...so this PR already fixes, then?

maybe add this add a test in that same routine?

```
        result = self.tsframe.replace(to_replace={nan: 0, 1: -1e8})
        result2 = self.tsframe.replace(to_replace=(1,nan),value=(-1e8,0))
        assert_frame_equal(result,result2)
.....

```

stop reading my mind!!!

:smile: 

its odd...ONLY came up on 2.6 on sparc (the dict iterates in a different order)....

once you merge this...I can retest on sparc and close that other issue

soon as this passes....good to merge ...

@jreback @jtratner ?

looks fine to me
ENH: json default_handler param
State: closed

This adds a `default_handler` param to `to_json`, which if passed must be a callable which takes one argument (the object to convert) and returns a serialisable object. The `default_handler` will only be used if an object cannot otherwise be serialised.

Note right now the JSON serialiser has direct handling for:
- Python basic types `bool`, `int`, `long`, `float`.
- Python containers `dict`, `list`, `tuple` and `set`.
- Python byte and unicode strings.
- Python decimal.
- Python `datetime` and `date`.
- Python `None`.
- Numpy arrays.
- Numpy scalars 
- `NaN` and `NaT`
- Pandas `Index`, `Series`, `DataFrame` and `Timestamp`
- any subclasses of the above.

If an object is not recognised the fallback behaviour with this PR would be:
1. if a `toDict` method is defined by the unrecognised object that 
   will be called and its returned `dict` will be JSON serialised.
2. if a `default_handler` has been passed to `to_json` that will
   be called to convert the object.
3. otherwise an attempt is made to convert the object to a `dict` by
   iterating over its attributes. 


Labels:

Comments:
didn't this have an original issue? (or was that on the time serializatino?)

if not, just reference this issue in the release notes

passed on windows?

thanks.....(I updated the release notes)...and confirmed windows...
ENH: Support for PyExcelerate as an Excel writer engine.
State: closed

This is an initial patch to add PyExcelerate as an Excel writer engine.

**It isn't suitable for merge** yet since there is issue with date handling in PyExcelerate that I am working to resolve with the authors. I just want this to appear on the pandas timeline for now so I can refer to it in other PRs.
1. The requirements files are missing since the PR will require a PyExcelerate update and release. Also, the `print_versions.py` entry.
2. There is also a failing test due to versioning.
3. There is currently no merged cell support.

I'll fix all these before the final PR request. 

John


Labels:

Comments:
@jmcnamara awesome - really interested in this!

can you give a use case where PyExcelerate would be used rather than XlswWriter (undertsanding that you are the author of XlsxWriter!)

@jreback Do you mean to put a use case in the docs?

The general use case for PyExcelerate would be that it is faster. In the optimised case it is 50% faster than XlsxWriter. (I don't know what the associated memory usage is, I'll have to benchmark it.)

The use case for XlsxWriter is that it is fast enough (as fast as xlwt) and that it supports a lot of features such as charts, conditional formatting, autofilters and more.

I can add a line or two about that to `install.rst` or `release.rst` if you think that would be useful.

@jmcnamara I think a brief explanation for all of the excel engines would be in order (in install.rst (or io.rst) with a link on the other side would be good). A user would be like, hmm, let me specify an engine if ..... (aside from the fact that the default engine order will do a good job)

@jreback Yes. Good point. I'll add that.

This is advocating that we incorporate a **fifth** optional-dependency for excel files support?!
That's sheer insanity. we should strive for less deps to do a given job, not more.

Given that it's also been gathering dust for 3 months, I'm just closing this.

> This is advocating that we incorporate a fifth optional-dependency for excel files support?!
> That's sheer insanity. we should strive for less deps to do a given job, not more.

I'm fine with closing it. Pyexcelerate isn't really stable enough or doesn't have enough formatting support for proper integration with Pandas.

That is also why the PR has stagnated for several months. I was waiting for the Pyexcelerate feature set to catch up.

All in all, it probably isn't worth adding at this stage.
add test for indexing with integer arrays
State: closed

I had a try at solving issue #5006, but it does not look as trivial to me as @jreback suggested. ;-)

I added a test for indexing with int-typed arrays containing single indices, which currently fails.  I spent some time reading test_indexing, because my hunch is that it could be better integrated with the existing tests (e.g. using check_result), but anyhow - any test is better than none, I guess.

The failing test goes through `_iLocIndexer._getitem_axis()` -> `_NDFrameIndexer._get_loc()` -> `Series._ixs()` -> `index.get_value_at()` -> `util.get_value_at()`, but I don't see the point where that should've been different right now.

Sorry, but I lack time for solving this at the moment.


Labels:

Comments:
Don't forget to setup Travis.

Thanks for the test case! (doesn't matter to setup Travis if it's already a failing test). One thing that could be helpful is changing the test to show exactly what Series you expect, but that's minor.

Would be nice to have a known fail or something to keep track.

closing in favor of #5134

@hmeine 

thanks for the test case. This is straightforward to fix; but a little bit non-intuitive when first looking at the code. There are many times when an exception is used to decide what happens, in this case a list causes the index engine to raise an InvalidIndexing error and then the series is ultimately reindexed (using the indicies as locations). The ndarray was actually returning the value directly (the theory goes that the list could actually be holding something else so best not to attempt a direct get using the cython code, while the ndarray can only hold indices so it is safe).

fixed up...thanks for the catch!

Ah, OK, I was not prepared for the exception trick. ;-)

I wonder however whether you plan to optimize the fix in the future – always copying (potentially large) arrays into lists seems to bring a potential performance regression, no?  I would have added a TODO or an `if len(key) > 1:` – potentially in cython – or something.

always profile.

You would be optimizing us, and a single copy is generally not problematic. In addition, you would generally fall back to boolean indexing if you are selecting a sizable number of indices as that is generally how you would be generating them. Your usecase is quite specific (using nonzero) and is an edge case.
BUG: Fix segfault on isnull(MultiIndex)
State: closed

Now raises NotImplementedError b/c not yet clear what it should return.

Fixes #5123.


Labels:

Comments:
cc @TomAugspurger 

I tried your branch and your fix worked.  Thanks.

okay, so any objections to raising NotImplementedError here?

no objections ... can u remove the unused `_test` import?

oh nevermind, wrong file

One benefit of changing MI representation to _not_ be a subclass of ndarray will be to avoid issues like these. I'm pretty sure there are other methods where this is an issue as well.

> One benefit of changing MI representation to not be a subclass of MultiIndex

Do u mean `Index`?

segfault is happening bc MI shape is 0 and a length 0 array is ultimately passed to `lib.isnullobj` which is not doing a manual bounds check before trying to access elements and it probably has the `@cython.boundscheck(False)` decorator enabled

why not just return a `MultiIndex` with `True`, `False`?

``` python
def _mi_is_null(mi):
    values = mi.values
    avalues = np.array(lmap(list, values), object)
    null = isnull(avalues)
    return MultiIndex.from_arrays(null, names=mi.names)
```

though the utility of this isn't so clear to me...why would you pass a `MultiIndex` to `isnull` in the first place? i agree with notimplementederror, but i also think at some point we should decide to return something and not raise

I meant ndarray.

@cpcloud can we save that for 0.14 instead of choosing what it should return now? It's sort of the same thing as bool on an NDFrame. It could either be True if _any_ value on any level at a location were NaN, or True only if _all_ were NaN at that point.

Also, the MultiIndex returned by `from_arrays` like that would be nearly unusable, given that it would just be a huge number of dups (only can take True or False  on each level). Better to just return the ndarray instead.

agree with all those points ... i think ndarray is a better return there

merge this?

I doubt has a perf impact but maybe give a quick test

okay, do you want to just leave this for 0.14 when MI won't be a subclass of ndarray and the problem won't exist (because we'll have to add a check to isnull regardless)?

no, I actually think this needs to be in 0.13 (at least for the moment to avoid the segfault). I am going to post a perf check soon (though as I said I don't think this is a problem....but anytime I change stuff in heavily used routines can't hurt to check)

okay - sorry to pass the buck, I just don't trust my machine to do perf tests well. If I get my hands on a non-laptop then I could trust it.

no..prob

other option is to do:

``` python
values = getattr(obj, 'values', obj)
```

and then add view(cls) at end, but you get more metadata issues and such.

this is fine perf wise....go ahead and merge

go ahead and merge this

@jtratner this mergeable, right?

Yeah, I just rebased it, hopefully it'll just pass.
TST: Tests and fix for unhandled data types in Excel writers.
State: closed

Added tests and a fix for unhandled numpy data types in the
Excel writers. Issue #3122.


Labels:

Comments:
Some background to this PR:
1. The original issue #3122 is due to the fact that xlwt whitelists the types that it handles and raises an exception for unhandled types. The fact that certain `numpy` types aren't identified correctly may be a `numpy` bug but their devs have rejected that as an issue.
2. The `pandas/io/excel.py` module contains a `_conv_value()` method to map pandas/numpy types to types handled by xlwt/openpyxl.
3. This PR extends/strengthens the `_conv_value()` function to handle more generic types.
4. This PR adds more extensive, although non-exhaustive, tests for `numpy` [data types](http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html).
5. XlwsWriter doesn't require the `_conv_value()` method since it has its own type conversion methods. So I've removed that from the `xlsxwriter` engine.

@jtratner ?

Going to take a look tonight or tomorrow.
On Oct 7, 2013 7:36 PM, "jreback" notifications@github.com wrote:

> @jtratner https://github.com/jtratner ?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5122#issuecomment-25854366
> .

@jtratner this ok for merging?

Looks good. 

Thanks for fixing this!

And :+1: for not doing `_conv_value` where it's not needed.
CLN: clean up parser options
State: closed

Also add a test to make sure that the C parser options validation is actually 
covered

example travis fail:

https://travis-ci.org/pydata/pandas/jobs/12171563

explanation (for the record):

if you print out `options` before this commit you'll see that they are all overwritten with a single value. Whatever this value is will depend on how the `_parser_defaults` dictionary happens to be ordered (which is random because of hashing). 

This caused a bug because the value of `value` is retained from the loop over `_parser_defaults`, and `default` is _not_ reassigned in the two loops that follow the loop over `_parser_defaults`, so all the values end up being the same as the last argument that was iterated over from `_parser_defaults`.

magic explained :tada: 


Labels:

Comments:
Great work catching this! :+1: 

i'm not sure about this factorize argument ... now that i think about it i think it hsould be removed from the signature and set of arguments that are incompatible with the python parser

@jreback @jtratner comments?

seems fine, any idea what factorize does?

nothing. wasn't used anywhere in `parser.pyx` except to be assigned as an instance variable never to be used again

looks like it was added as an option in c4d36ca5681892e506d58125ba394caf2135d5ad and has just been floating around since then doing nothing
CLN: Remove redundant initialization from Int64Index and Float64Index constructor
State: closed

`Int64Index` repeated itself and actually reconstructed an ndarray twice when passed in data was an iterable. This simplifies it down but does not actually change any behavior. (you'll see that there was basically exactly the same code twice)


Labels:

Comments:
this looks ok

@jreback changes Float64Index too - okay?

prob was the same issue
DOC: doc fixups
State: closed

closes #5110.


Labels:

Comments:
@cpcloud 

Seems like the is an issue with the example in the documentation.  Can you reproduce the rng_hourly_eastern.values below?  Clearly it's wrong, but I cannot reproduce locally.

```
In [177]: rng_hourly = DatetimeIndex(['11/06/2011 00:00', '11/06/2011 01:00',
   .....:                             '11/06/2011 01:00', '11/06/2011 02:00',
   .....:                             '11/06/2011 03:00'])
   .....: 

In [179]: rng_hourly_eastern = rng_hourly.tz_localize('US/Eastern', infer_dst=True)

In [180]: rng_hourly_eastern.values

array([1970-01-16 164:00:00, 1970-01-16 165:00:00, 1970-01-16 166:00:00,
       1970-01-16 167:00:00, 1970-01-16 168:00:00], dtype=datetime64[ns])
```
BUG: allow plot, boxplot, hist and completion on GroupBy objects
State: closed

closes #5102 

also fixes extra plots created by groupby plotting (couldn't find an issue), now there's a test that ensures that groupby only plots the number of axes/figures it's supposed to
removes an unused internal groupby parameter


Labels:

Comments:
@jreback @jtratner thoughts?

should those plot tests be marked slow?

alright this is ready to go pending review, @jreback @jtratner ?

good

maybe put a tab complete example for say frames in docs as well (basics?) later if u can

done ... i put in `10min.rst` since tab completion is so useful that people should know about it as soon as they start using pandas.

gre8...looks fine to me

@nehalecky there you go :)

Grand, thank you so much @cpcloud, this works perfectly! :)
TST: win32 paths cannot be turned into URLs by prefixing them with "file://" v2
State: closed

Rebased on master (unable to use same pull request). Closes #4580.

see http://stackoverflow.com/questions/11687478/convert-a-filename-to-a-file-url


Labels:

Comments:
@cpcloud for 0.13?

@cpcloud 0.13?

@mindw Can you rebase? We'd like to get this in for 0.13 (by next week). Thanks!

rebase to where?

Rebase to this PR.

Assuming you're on the local branch associated with this PR:

``` sh
# add pandas upstream if you don't have it already
git remote add upstream git://github.com/pydata/pandas.git

# fetch upstream changes (do this anyway)
git fetch upstream

# rebase on top of upstream/master (this is current master)
git rebase upstream/master

# force push
git push --force
```

You might need to resolve merge conflicts before force pushing.

ping.

@cpcloud pls review this, or can defer to 0.14....up2u

bumping to 0.14, unless reviewed before

I'll take it.

I'd prefer the io/common name to be `file_path_to_url` - seems clearer. Otherwise should be fine, so long as it passes Travis.

Also please rebase again and squash your commits together. 

Ok, please review.

Why was it pushed to 0.14 again?

Ok.

this is fine, just remove the line.

I can't merge this until you enable Travis though - looks like you had it set up months ago - see more here - https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md#steps-to-enable-travis-ci

I disabled it due to known test failures (test_fred?)

those are fixed now, but can't merge this unless you set up Travis.

Ok fixed. Waiting in line.

-gabi

On Tue, Oct 29, 2013 at 12:13 AM, Jeff Tratner notifications@github.comwrote:

> those are fixed now, but can't merge this unless you set up Travis.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5100#issuecomment-27262050
> .

thanks!