BrandonFors/pandas-github-issues huggingface dataset:
DOC: isin Example for .13 release notes
State: closed

I also fixed a few typos in a separate commit.  I can remove that commit if they're going to cause conflicts.

Also did I get the syntax correct for this link? 

> See :ref:`Boolean Indexing<indexing.boolean>` for more


Labels:

Comments:
Okay, I moved the Series and DataFrame isin sections of `indexing.rst` into their own subsection under Basics.

So pending a decision on how to handle #4421, which I don't have time to get to today, I think this is ready to go.

Off topic: @jreback were you aware that the `to_datetime` examples in What's New are failing?  http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#enhancements

I wanted to check before making an issue.

@TomAugspurger you mean the `to_timedelta` ones.....do you see that big warning about requiring numpy 1.7, unfortunately the build machine (which @changhiskhan / @wesm) run runs 1.6.2 :(.

timedeltas are so broken in 1.6 that only very limited ops can be done

Added the Series example.

perfect..thanks

@TomAugspurger can your rebase this and can get in

@jreback rebased… but the typo fixes I made in my [second commit](https://github.com/TomAugspurger/pandas/commit/ad2986ec48bfc650b097e01f18caa268ea4038b6) got tangled up with some changes someone (probably you) made to the HDF5 notes.  Lines 35-109 my PR.  Should I go back to before my rebase and do it again with just the changes in my first commit?  There's a way to select which commits get applied right?

`git rebase -i upstream/master` should work

Ok, should be good to go.

gre8 thanks

@TomAugspurger @hayd I slightly changed the isin example in v0.13.0 (because it got moved and df was not defined correctly). Pls verify it and change if needed.

Looks good.
ENH: Added xlsxwriter as an ExcelWriter option.
State: closed

Added xlsxwriter as an optional writer engine. Issue #4542.


Labels:

Comments:
Merged comments from #4739 but created new PR based on @jtratner's refactoring of `excel.py`.

Note, this PR doesn't include test cases due to fact that `test_excel.py` has also been heavily refactored but isn't yet merged (#4844). Once, those cases are merged to master I'll rebase and update this PR. I've added `test_excel.py` tests on a separate branch and they pass local and TravisCI tests.

@jmcnamara I merged the other branch,

btw - if you need to change the ordering of how the cells are formatted (e.g., so that it yields cells in a different order or something), you could move the cell formatting to a method on `ExcelWriter` and then overwrite that in `XlsxwriterWriter` (or whatever you are calling it). I recall you saying something about that in regards to performance. (and then just pass additional keyword arguments to `write_cells`, etc.)

I can't seem to fix this cleanly. I'm going to close the PR, rebase and squash the commits.

What's the current status of other engines for excel?

I made an attempt to add pyexcelerate as a writer engine in PR #5128.

That stalled however and was eventually rejected. See the comments at the end of the PR for the reasons.

Apart from that, there aren't currently, to the best of my knowledge, any Excel reader/writers that would be worth adding.
BUG: fix sort_index with one col and ascending list
State: closed

Fixes #4839.

now actually checks the first element of the list


Labels:

Comments:
@jburroni this should fix your issue - can you try it?

It is working.

@jburroni okay, I'm glad I could fix this quickly. I will merge this in the next day or so.

Thanks for reporting this!

Thank you for fixing it so fast

@jtratner I think you need a test case where the len of the ascending list is not the same as the length of the sort keys (not sure what happens now), e.g. it should be == in length, or a scalar

agree!

On Mon, Sep 16, 2013 at 10:06 AM, jreback notifications@github.com wrote:

> @jtratner https://github.com/jtratner I think you need a test case
> where the len of the ascending list is not the same as the length of the
> sort keys (not sure what happens now), e.g. it should be == in length, or a
> scalar
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4846#issuecomment-24508071
> .

## 

" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )
ENH: Better read_json error when handling bad keys
State: closed

(in json data with orient=split)

Fixes #4730.


Labels:

Comments:
@Komnomnomnom does this look okay to you? Just changes the exception to be slightly clearer...

E.g.

``` python
import pandas as pd
from pandas.compat import StringIO
import json
d = {u'columns': [0],
 u'data': [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9]],
 u'index': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
 u'badkey': [5]}
s = StringIO(json.dumps(d))
pd.read_json(s, orient='split')
```

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/io/json.py", line 182, in read_json
    date_unit).parse()
  File "pandas/io/json.py", line 242, in parse
    self._parse_no_numpy()
  File "pandas/io/json.py", line 478, in _parse_no_numpy
    self.obj = _run_with_kwarg_exception(DataFrame, decoded, dtype=None)
  File "pandas/io/json.py", line 391, in _run_with_kwarg_exception
    raise ValueError(msg)
ValueError: JSON data had unexpected key: 'badkey'
```

and I think it's clear enough in Python 3 too:

```
Traceback (most recent call last):
  File "./pandas/io/json.py", line 382, in _run_with_kwarg_exception
    return func(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'badkey'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./pandas/io/json.py", line 182, in read_json
    date_unit).parse()
  File "./pandas/io/json.py", line 242, in parse
    self._parse_no_numpy()
  File "./pandas/io/json.py", line 478, in _parse_no_numpy
    self.obj = _run_with_kwarg_exception(DataFrame, decoded, dtype=None)
  File "./pandas/io/json.py", line 391, in _run_with_kwarg_exception
    raise ValueError(msg)
ValueError: JSON data had unexpected key: 'badkey'
```

I agree that the exception is much clearer. Rather than relying on the text of the message in the raised `TypeError`, which seems kind of fragile, would it be better to instead check that the decoded keys are only those encoded by `split` and raise a `ValueError` directly if not?

`columns`, `index`, `data` for DataFrame
`name`, `index`, `data` for Series

@Komnomnomnom thanks for the cleaner matchgroup usage. I did it this way only to make it easier to maintain...but I guess we can just define what the keys are in json once, so no big problem.

@Komnomnomnom any other elements need guards for bad keys? Otherwise, this is probably good.

Nope `split` should be the only orient using named keys. Thanks @jtratner!
io.pytables: handle start/stop in remove and select_coords
State: closed

I realized that the start/stop arguments to HDFStore.remove are ignored, so I have a patch to make them work as expected. Moreover I changed Selection.select_coords to also apply start and stop.


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
need:
- hook up to travis
- tests for new behavior (can go with test_remove, or create a new sub-below up2u)
- release notes

thanks!

Thanks for your feedback. I can do this next this week. What exactly is needed to "hook up to travis", I did not find anything about this in the [deveolpoer page](http://pandas.pydata.org/developers.html).

look at contributing.md in main pandas dir

@cpcloud maybe add a link on developer page?

@jreback @cpcloud I'd suggest copying the part on Travis wholesale.

done: https://github.com/pydata/pandas-website/pull/24

@wabu how's this coming?

sorry. caught a bad cold, now added missing bits

no problem!

fyi...you will need to rebase that master out of their

`git rebase -i upstream/master`

@wabu can you rebase this?

I did a rebase, the test is still failing on python 2.7, did not have time to debug this as i don't have a numpy/pytables/pandas for 2.7 around

Travis builds the pr automatically

https://travis-ci.org/pydata/pandas/jobs/17820546

look at #6177 for fixes
BUG/VIS: correctly test for yaxis ticklocs across different versions of MPL
State: closed

closes #4789


Labels:

Comments:
i'm going to add a wheel for mpl 1.3 on travis...will run on the py27 "not slow" job

@jreback @y-p any objections to that?

sounds good

actually it has to run on the "slow" build

i tried this with matplotlib 1.1.1, 1.2.0, 1.2.1, and 1.3.0 locally and tests pass so if there aren't any other objections... going to merge in a bit

we only have 1 plot build? or 2?

just 1 bc of the slow dec....there are a few plotting tests that aren't marked  slow

ok 2 merge?

do we have another plot build? (or just the one)?

oh...sorry. ou answered that....add another of the builds?

actually its ok...nvm
TST: Add test case for 0.12 regression with loc from GH4825
State: closed

Closes #4825.


Labels:

Comments:
thanks
let me see where this is coming from in any event

@jreback btw, should this raise?

``` python
ser = Series([0.2, 1, 2])
ser.loc[[3, 3]]
```

@jreback so I think this is unnecessary now, right?

yep I incorporated them (slightly modified)
BUG: Fix copy s.t. it always copies index/columns.
State: closed

Fixes #4202 (and maybe some others too).

Only copies index/columns with `deep=True` on `BlockManager`. Plus some 
tests...yay!


Labels:

Comments:
@hayd copying will work now :)

this works for everything but SparseSeries and SparsePanel at this point...still trying to figure them out.

re identical 

u might try something like

if a is b or a.base is b.base: ( maybe do a share memory check too)
   return True

return a.identizal(b)

so the actual identity  case is a lot faster

Problem is that bases gets changed with view.  I.e.:

```
In [2]: i = Index(range(10)
   ...: )

In [3]: i.base
Out[3]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

In [4]: i.view().base
Out[4]: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64)

In [5]: i.view().view().base is i
Out[5]: True
```

so slight tweak on base.

@jreback @cpcloud here's what I'm thinking for index, adding an `is_` method:

``` python
    def is_(self, other):
        """returns True if and only if the indices actually are the same underlying array.
         Generally survives through views. Doesn't care about metadata."""
        if self is other:
            return True
        s_base, o_base = self.base, other.base
        return self is o_base or other is s_base or s_base is o_base
```

(base calls aren't cached, which is why it needs to be setup this way)

blah, this doesn't quite work either :-/ multiple levels of view screw up `.base`.

It's going to take some thinking to decide on how best to efficiently handle flexible metadata for `Index`. I was thinking of setting an `_id` attribute that gets copied on views but changed when slicing, copying or setting attributes. then the `is_` check would just be comparing the equality of two ids (probably hashes of id + type name).

Got this working (and faster) with the `is_` method from #4909. @cpcloud you want to take a look at either this or #4909 to see whether you think this is fast enough for groupby?

i'll vbench it and report back :)

if it's not good enough, we can shave off about 600-800ns per iteration by skipping the function call and using `self.index._id is other.index._id`

Is this the right behavior?

```
ind = Index(range(10))
df = DataFrame(range(10), ind)
assert df.index is ind
assert (df + 1).index is ind
```

@jtratner yep

@jreback @cpcloud does this look okay to merge?

ok

yep
ENH/API: allow DataFrame constructor to better accept list-like collections (GH3783,GH4297)
State: closed

closes #3783
closes #4297
related to #2305 (as now accept GeneratorType), but just convert to list, don't incrementally create the frame


Labels:

Comments:
@jtratner that worked....now not dependent on the type per se only that its list-like and 1dim

@jtratner @cpcloud ?

:+1:
PERF: much faster to_datetime performance with a format of '%Y%m%d'
State: closed

```
In [1]: rng = date_range('1/1/2000', periods=10000, freq='D')

In [2]: strings = Series(rng.year*10000+rng.month*100+rng.day,dtype=np.int64).apply(str)

In [3]: %timeit pandas.tslib.array_strptime(strings.values,"%Y%m%d")
10 loops, best of 3: 42.9 ms per loop

In [4]: %timeit pd.to_datetime(strings,format="%Y%m%d")
100 loops, best of 3: 9.21 ms per loop

In [5]: (pd.to_datetime(strings,format="%Y%m%d") == pandas.tslib.array_strptime(strings.values,"%Y%m%d")).all()
Out[5]: True
```


Labels:

Comments:
Does this apply to read_csv as well? I don't think it calls into to_datetime, does it?
we should get our fast paths consistent everywhere

I don't believe so, this should be wrapped up in `tslib.array_to_datetime`  / `tslib.try_parse_dates`

or prob better is to have `read_csv` use `to_datetime` (or a related call) directly; the conversions are done after parsing anyhow (so all python and not cython code)

so maybe fix to_datetime (in #5490) and then have another issue to wrap back into read_csv...

Sounds good, open an issue for 0.14?

Added note: 433ddc3
... and fixed the typos: 38e3b90. _sigh_

yep
ENH/CLN: support enhanced timedelta64 operations/conversions
State: closed

closes .#3577
realted #3009 
- ENH: add top-level `to_timedelta` to convert string/integer to timedeltas
- TST: add pandas/tseries/tests/test_timedeltas.py
- API: add full timedelta parsing and conversion to np.timedelta64[ns]
- CLN: refactored locations of timedeltas to core/tseries/timedeltas (from a series of functions in core/common)
- ENH: support timedelta64[ns] as a serialization type in HDFStore for query and append (GH3577)


Labels:

Comments:
@cpcloud @jtratner comments?

@cpcloud this ended up morphing...but now have `to_timedelta` top-level function....pretty nice if I do say so myself!

All of the experimentalists using pandas will now be chomping at the bit for `TimedeltaIndex`. You've teased them with a `to_timedelta`!

One does not simply ....

implement `to_timedelta` without `TimedeltaIndex`

:smile: 

hah...I just returned a Series, easier (and pretty close actually)

`TimedeltaIndex` is for you!
ENH: Allow abs to work with PandasObjects
State: closed

Add `__abs__` method so that you can use the toplevel `abs()` with
PandasObjects. I note there aren't that many existing test cases for
abs. At some point we might want to think about that.

Also related #4819 

cc @thisch


Labels:

Comments:
if no objections, will merge this soon.  Entirety of the change is adding this:

```
def __abs__(self):
    return self.abs()
```

this is fine....just an oversight, I think < 0.13, `abs(a_series)` worked because numpy defined **abs**

Yep numpy does define abs. Looks good. 

thanks for pointing this out @thisch - now all pandas objects support abs and your original code should now work.

what the heck? I added release notes to this and pushed them to Github before merging, but they didn't end up in this PR. weird. Worth committing separately @jreback ?

ha....just was noticing that...no...you can throw them in next time you merge something

no test for poor-old `Series` :cry:?

It gets tested in test_panel (weirdly). If we want to add abs tests, we should do it to test the abs method...this is basically just adding some sugar :)

ah ok yep i c now
BUG: Fix read_fwf with compressed files.
State: closed

Fixes #3963.

`gzip` and `bz2` both now return `bytes` rather than `str` in Python 3, so
need to check for bytes and decode as necessary.

replacing #4783


Labels:

Comments:
@y-p this look okay to you now?

You already have a +1, what more do you want, blood? :)

reopened #3963. bad +1 by me
BUG: Fix input bytes conversion in Py3 to return str
State: closed

Fixes #3963, #4785

Fixed bug with reading compressed files in as `bytes` (`gzip` and `bz2` both now return `bytes` rather
than `str` in Python 3) rather than `str` in Python 3, as well as the lack of conversion of `BytesIO`. Now, `_get_handle` and `_wrap_compressed` both wrap in an `io.TextIOWrapper`, so that the parsers work internally only with `str` in Python 3. In Python 3.2, has to read the entire file in first (because `gzip` and `bz2` files both lack a `read1()` method in 3.2)

Also adds support for passing fileobjects with compression == 'bz2'.


Labels:

Comments:
bz2 in python 3.2 doesn't support a `read1` method, for what look to be historical reasons. It does support `read1` in Python 3.3. Because it's a C module, you can't monkey-patch it directly.

@y-p I'm getting closer on this, but decided to give up and just read the entire file in with 3.2, then decode and rewrap with StringIO -- otherwise we couldn't support compressed files (easily) in that version...

Yep, seems like a reasonable minimal workaround to me.

@y-p look okay to you? going to merge if so...

Yep, go ahead.
ENH: Add axis and level keywords to where, so that the other argument can now be an alignable pandas object.
State: closed

So traditionally a fillna that does the means of the columns is an apply operation

```
In [1]: df = DataFrame(np.random.randn(10,3))

In [2]: df.iloc[3:5,0] = np.nan

In [3]: df.iloc[4:6,1] = np.nan

In [4]: df.iloc[5:8,2] = np.nan

In [5]: df
Out[5]: 
          0         1         2
0  0.096030  0.197451  1.645981
1 -0.443437  0.359204 -0.382563
2  0.613981  1.418754 -0.589935
3  0.000000  0.449953 -0.308414
4  0.000000  0.000000 -0.471054
5 -2.350309  0.000000  0.000000
6 -0.218522  0.498207  0.000000
7  0.478238  0.399154  0.000000
8  0.895854  0.230992  0.025799
9  0.085675  2.189373 -0.946990
```

The following currently fails in 0.12 as where is finicky about how it broadcasts

```
In [4]: df.where(df>0,df[0],axis='index')
ValueError: other must be the same shape as self when an ndarray
```

Adding `axis` and `level` arguments to `where` (which uses align under the hood), now enables the `other` object to be a Series/DataFrame (as well as a scalar) without a whole bunch of alignment/broadcasting. This also should be quite a bit faster.

```
IIn [6]: df.where(df>0,df[0],axis='index')
Out[6]: 
          0         1         2
0  0.096030  0.197451  1.645981
1 -0.443437  0.359204 -0.443437
2  0.613981  1.418754  0.613981
3  0.000000  0.449953  0.000000
4  0.000000  0.000000  0.000000
5 -2.350309 -2.350309 -2.350309
6 -0.218522  0.498207 -0.218522
7  0.478238  0.399154  0.478238
8  0.895854  0.230992  0.025799
9  0.085675  2.189373  0.085675
```

This works in 0.12.

```
In [7]: df.apply(lambda x, y: x.where(x>0,y), y=df[0])
Out[7]: 
          0         1         2
0  0.096030  0.197451  1.645981
1 -0.443437  0.359204 -0.443437
2  0.613981  1.418754  0.613981
3  0.000000  0.449953  0.000000
4  0.000000  0.000000  0.000000
5 -2.350309 -2.350309 -2.350309
6 -0.218522  0.498207 -0.218522
7  0.478238  0.399154  0.478238
8  0.895854  0.230992  0.025799
9  0.085675  2.189373  0.085675
```


Labels:

Comments:
`fillna` works with a `Series`, see my answer on [SO](http://stackoverflow.com/a/18691949).

@bmu right....I had enabled that for 0.12 (duh!)
(and I fixed the doc string)

should have had a better example one that is somewhat non-trivial

I have updated the top display
DOC: correction of example in unstack docstring
State: closed

There is an error in the example of the `unstack` docstring (http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.unstack.html ):

```
>>> df = s.unstack(level=0)
>>> df
   one  two
a  1.   2.
b  3.   4
```

should be 

```
   one  two
a  1.   3.
b  2.   4
```

So this PR fixes that, but in the same time I did some other adjustments. Mainly:
- added a "see also"
- added in the example the code to create the series, so one can run the example him/herself


Labels:

Comments:
@cpcloud OK, whitespace removed!

@cpcloud okay to merge?

yep

thanks @jorisvandenbossche !
CLN: Move print_versions into pandas proper.
State: closed

Moves print_versions under util so that it can be called from buildbot
(and so we can tell people to run `from pandas.util.print_versions import show_versions; show_versions()`
when trying to debug)

cc @yarikoptic


Labels:

Comments:
Any objections? This is minor...

fine by me

I checked that running `python print_versions.py` works and this runs appropriately on Travis [i.e. it prints versions]

@yarikoptic - ask and ye shall receive, you can use:

``` python
from pandas.util.print_versions import show_versions
show_versions()
```
BUG/ER:  (GH2702) Bug with Series indexing not raising an error when the right-hand-side has an incorrect length
State: closed

closes #2702

Base use cases (note that a completely changed series will morph to the new dtype if necessary)

```
In [1]: s = Series(list('abc'))
In [2]: s[0:3] = list(range(3))

In [3]: s
Out[3]: 
0    0
1    1
2    2
dtype: int64

In [4]: s = Series(list('abc'))

In [5]: s[0:2] = list(range(2))

In [6]: s
Out[6]: 
0    0
1    1
2    c
dtype: object
```

This is a bit odd, but allowed

```
In [4]: s[0] = list(range(10))

In [5]: s
Out[5]: 
0    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1                                 b
2                                 c
dtype: object
```

These all will raise error messages now

```
In [1]: s = Series(list('abc'))

In [2]: s[0:3] = list(range(27))
ValueError: cannot set using a slice indexer with a different length than the value

In [3]: s[[0,1,2]] = list(range(27))
ValueError: cannot set using a list-like indexer with a different length than the value

In [4]: s[[0,1,2]] = list(range(2))
ValueError: cannot set using a list-like indexer with a different length than the value

```

With a step and a slice indexer (not sure was tested before)

When the length of the slicer matches exactly it works

```
In [5]: s = Series(list('abcdef'))

In [6]: s[0:4:2] = list(range(2))

In [7]: s
Out[7]: 
0    0
1    b
2    1
3    d
4    e
5    f
dtype: object
```

Raises when it doesn't

```
In [8]: s[0:4:2] = list(range(27))
ValueError: cannot set using a slice indexer with a different length than the value

```


Labels:

Comments:
@cpcloud @jtratner pls have a look when you can

there are some in test_frame
Added support for outer join in DataFrame.update
State: closed


Labels:

Comments:
Can you add a test and hook up travis-ci? thanks

@dougvk you should rebase this on top of master first (an additional test case for the Exception was added here plus a change to error message).

@dougvk need:
- tests
- hook up to travis

whoops, completely forgot. will get on this

@dougvk ? tests and pls hook up to travis

Stalled.