BrandonFors/pandas-github-issues huggingface dataset:
Fix rename for MultiIndex columns DataFrame
State: closed

DataFrame rename doesn't rename a MultiIndex column and it flattens the columns. Closes https://github.com/pydata/pandas/issues/3165 which is a fix for index but not columns.


Labels:

Comments:
can u hook up to Travis?
instructions in contributing.md in main dir

also pls add a release notes mention

Okay I think I've hooked it up to Travis. Do I have to run manual build? 

push another commit or

git commit --amend -C HEAD (will reset the latest coit hash)

then 

git push yourfork yourbranch -f

then Travis will build it

Should I add the release note mention here? doc/source/v0.11.1.txt

release notes in doc/source/release.rst

ok...looks fine, except need to rebase, don't want to have those `Merge branch` in there....

```
git checkout yourbranch
git rebase -i
```

and remove the offending commit (just delete it) 05bd8618ab21701f7067969e84c9200c88ed4848

also prob want to `s` squash commits together

then

```
git push yourfork yourbranch -f
```

will rewrite this branch

Sorry I'm still learning git, I hope I did your last instruction right, but I'm lost with deleting the commit Merge branch. 
When I do git rebase -i the offending merge commit is not even there, I just proceeded with squashing, and after saving  git log doesn't show the merge commit anymore.

Thanks for the fix, I ran into the same problem today and was about to start on a fix but you beat me to it.    :)

I have tested your code and it works as expected for me, matching the functionality for indexes already put in by 3165.

Let me know if there is anything I can do to help.

@ogiaquino it looks like you rebased successfully, good work!
CLN: clean up data.py
State: closed

closes #4001 
closes #3982
closes #4028


Labels:

Comments:
everything still work? (not sure how many tests we actually have on io.data)

can u squash the num of commits down a bit?

@jreback yes everything still works, but not quite finished yet...will squash when i check off the list items

the fact that everything still works after all these changes makes me think that there is potentially a lot of untested code in `data.py`...

@cpcloud I agree with you on that. Do you know why the main entry point is named like a class (`DataReader`) but isn't really a class? It's confusing because the functions seem really similar and a good candidate for abstracting into a class definition. Not even necessarily because it would reduce code duplication, but because it could make the file easier to follow (i.e., a workflow described as `get_data` -> `clean_data` -> `package_data [into appropriate type, like Series/Panel]`) and would allow us to do a better job in standardizing error handling and making it easier to add in new data sources (or allow people to make plugins or something).

Would it make sense to set a default timeout on these requests? (you can pass a `timeout` keyword to urlopen). Helps the test suite and gives more flexibility to users. 

possibly...somewhat related is that there's a `retry_count` all over the place which i think may be unnecessary and might overlap a bit with `timeout`...timeout might be nice although i'm not sure when you'd want to use it. maybe u have some specific use cases in mind?

@jreback on this clean up i have those tests that fail skip if they raise an `AttributeError`. i feel a little weird about that .... how should i handle this error, seems hard to predict since lxml will return but then not have any html...how to test for that? could return `nan` if that happens or raise...

I would have an exception fail them. I guess you can't really 'validate' these consistently. I think these reutrn a list, so a len(0) list is valid then; only complete the rest of the test if you have len > 0. Maybe I would print a warning (in the test) if this is the case, so that when running in the command line, we know this has 'failed'.

AssertionWarning? a little bit cheesy...but what can you do

hold on maybe there's something different between using expiry vs passing month/year explicitly. only the warning tests seem to be failing

nope that looks ok

after this passes i'm going to do one more rehash and push then im gonna merge

going to also run tox a few times in a row to see if i can get the warnings to show up

@jreback any objections here?

kind of a big overhaul, but badly needed 

looks good

round 42 on this...here we go.

I picked a hell of a time to decide to make a minor change to data.py! Things are moving quickly :+1: I'll have to check back on #4044 once the dust has settled

sorry man there's just a bunch of stuff that needed to essentially be rewritten and network stuff cleared up (still some issues there)
BUG: invert_xaxis (negative tot_sec) triggers MilliSecondLocator (#3990)
State: closed

Solves issue #3990. 
Negative timedelta for xaxis range will no longer trigger MilliSecondLocator.


Labels:

Comments:
Do you think this is a correct fix?
And do I have to include some more things (test, mention in release notes, ..)

didn't see this....does this resolve the issue?

@cpcloud can you take a look?

there must be a reason that they are negative

@jorisvandenbossche can you explain why this fixes the problem? the `relativedelta` is allowed to be negative, but on the other hand, maybe whoever wrote this didn't take that into account.

e.g.

```
In [19]: a,b=date_range('20010101', periods=2)

In [20]: from dateutil.relativedelta import relativedelta

In [21]: d=relativedelta(a,b)

In [22]: delta = d

In [23]: paste
        num_days = ((delta.years * 12.0) + delta.months * 31.0) + delta.days
        num_sec = (delta.hours * 60.0 + delta.minutes) * 60.0 + delta.seconds
        tot_sec = num_days * 86400. + num_sec
## -- End pasted text --

In [24]: tot_sec
Out[24]: -86400.0
```

that doesn't like a case of "i forgot to `abs`", i could be wrong though.

For me this resolves the issue (but I submitted the patch of course :-) Although I don't really know if I could write a test for it.

`relativedelta` is indeed allowed to be negative, and **is** negative in the case of an inversed xaxis. That is not the problem. However, when deciding which tick Locator we want to use, it doesn't matter if it is inversed or not: we are interested in the absolute time delta. Because now the negative delta triggers a MilliSecondLocator just because it is negative, while the time delta is not suited for Milliseconds.

Some more details with references to code:
When using ax.invert_xaxis() the left and right xlim are interchanged. Because of this the delta in PandasAutoDateLocator will be negative (https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L241).
This in turn triggers MilliSecondLocator instead of the normal AutoDateLocator (because `tot_sec` in line https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L247 is negative), and causes the error as the datetime range is too big for MilliSecondLocator.

makes sense

@jreback i think this is ok.

@jorisvandenbossche thanks for the explanation!

ok..
@jorisvandenbossche can you add a quick release note?

OK, will do when I get home.

Thanks for the look at it!

I updated the commit with release notes. So for me it is ready to merge.

As I said in the issue, the only thing I don't understand is why the bug does not appear when using pandas plot method to create the axes object (so `ax.plot(df.index, df['col'].values)` vs `df.plot(ax=ax)`)

ok....merging...thansk...keep us apprised if there are any addtl issues, or if you figure out the above (you can't open in issue if you want)
ENH: raise a more a useful exception on empty files
State: closed


Labels:

Comments:
@jreback any last words before i merge this? the fix is that the heuristic for table sizes assumes that there are columns and then if there aren't it will divide by zero so i catch and give a valueerror with a more useful msg.

this seems very specific
what does it do now wih your example?

raise a valueerror telling the user that they've given a file with no columns

better to catch a more specific exception than to hide a bug in the parser with `except:` no?

why is that bad? seems clear to me

why is this a bug if it raises a reasonable exception?

no i added that

it raises zerodivisionerror before this 

the fact that there's a heuristic in `parser.pyx` that could possibly divide by zero is not something a user should have to deal with...right?

sorry by now you mean before this pr i thought u meant after it

oh i c

can this be caught lower down (eg detected in tr parser)

sure should i do that

there's another possible issue here that i hvaen't tried let me check it out

iirc this is easiky detectable in parser.pyx
(you can raise the same error that u out here)
but it only catches in this circumstance so it's narrower (and if something else is hiding will see that separately)

@jreback i think this is ready 2 go if u think so

looks good
BUG: Index shift drops index name
State: closed

```
>>> idx = pd.Index([pd.to_datetime('2013-06-21')], name='idx')
>>> idx.name
'idx'
>>> idx.shift(1, freq='B').name == None
True
```


Labels:

Comments:
looks good....

can you add a release notes mention (they are now in doc/source/release.rst) FYI
(since it doesn't have an issue, just put in the note, in the bug fix section)

thxs

Related?:

```
from pandas import Series, Index
s = Series(range(3), name='name')
Index(s).name
```

Returns None

@joeb1415 

you have to pass it in the constructor, e.g.

`Index(s,name=s.name)`

it could be automatic, but not sure of a use case for this

Release notes added.

@mtkni  Another pesky name bugs down, good work! :)
DOC: put release notes link to dev until 0.11.1 is released
State: closed


Labels:

Comments:
@jreback can i merge this? i just tried the new docs and dev works fine has release.html etc but since the release isn't out yet probably best just to link to dev docs. i will change back once 0.11.1 is released. i made a note to selves issue about it

ok.....fyi...docs are on public site...looks good!

cool thanks!
ENH: print more detailed truncated sequence if a limit is given
State: closed

works similar to numpy except that the edgeitems is overriden by threshold rather than the other way around which is what np does.

closes #3391 
- [x] add release notes


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@y-p can u take a look-see please

AH HA! i figured out the case where the overlap happens now i can write a test :) thanks @y-p for pointing that out

@y-p 

numpy's `edgeitems` (a new option i'm introducing called `max_edge_items`) overrides `threshold` (equivalent to our `max_seq_items`), when `threshold < edgeitems`.

i don't think numpy's behavior in this regard is necessarily what a user would expect. personally, i would expect `max_seq_items` to be a "hard limit" on the number of items that `repr` _including_ the items `repr`'d on the edge of the sequence, thus i think `max_seq_items` should override `max_edge_items`. the `edgeitems` would then be forced to be _at most_ `n / 2` of the sequence, guaranteeing no overlap.

what do you think? this, of course, will not be consistent with numpy. right now the above described behavior is what i'm doing, but i can change if consistency with numpy is preferred.

here's an example from the tests i've added:

``` python
with cf.option_context('display.max_seq_items', 2, 'display.max_edge_items', 5):
    s = 'Int64Index([0, ..., 9], dtype=int64)'
    res = repr(tm.makeIntIndex(10))
    assert_equal(s, res)
```

I agree. when I suggested aligning with numpy I meant the format for summary: "foo foo foo ... foo foo foo", i.e.
symmetric. what you propose sounds right to me. 

As simple as possible but no simpler.

for the special case of repring a `MultiIndex` when `max_seq_items == 1` i think the repr should be 

```
MultiIndex
[(u'C_l0_g0', ...), ...]
```

Put constraints on the setting of of max_seq_items (even number >=2), and save yourself the special cases.
Yes, for MultiIndex, considering each tuple as an item makes sense.

that certainly simplifies my life a lot

@y-p  can u take a look when u get a chance? thanks

@cpcloud close/push to 0.14?

@cpcloud ?

@cpcloud ?

pushing to 0.14

@y-p must have been accident

It's stale, but it needs to happen for 0.14. If @cpcloud doesn't finish, I'll do it.

@y-p gr8...i think i got all the accid closed ones...weird

@cpcloud what's the state of this?

I'll try to get this cleaned up this weekend.

Something happened just now when I tried to update the remote .... getting the infinity commits again.

Oh fudge I had those in a stash that has since been deleted .... ah well ... nothing major anyway. I'll open another PR.
CLN: Change bare exceptions pt 1
State: closed

Relates to #3954. In addition to changing exceptions types returned to be more explicit, this pull incorporates the following changes:
1. Adds a new `assertRaisesRegexp` to `util/testing.py` to port the `assertRaisesRegexp` helper from 2.7+ unittest
2. Cleans up stats/common.
3. Fix up initial assertions in `tseries.offset._cached_range` that were all off + fix the test cases which were all just raising TypeErrors because they were calling with the wrong arguments.
4. Changes the example timezone from Asia/Beijing to Asia/Hong+Kong b/c `Asia/Beijing` is not supported by pytz.
5. Any tz-aware : tz-naive comparison fails with TypeError, as will mismatched, localize, etc. calls.
6. Changes `SparseArray` indexing error messages to match tuple message for completeness.
7. Improves the window check in ols.

After you all say it's okay to merge, I'll update the docs to reflect changes.


Labels:

Comments:
ready?

If you're okay with the extra fixes (cached_range, stats/common, etc) then yes it's ready

looks good

@wesm ?

Fine with me

@jtratner merging.......3.2.1.
ENH: add last element to repring of sequences
State: closed


Labels:

Comments:
the mutual recursion should bottom out in `_pprint_seq` during either the call to `next` (will raise `StopIteration`) or it will raise `StopIteration` when `i not in range(min(nitems, len(seq)))`. i will also change that to `xrange` because that will be a memory hog for large sequences with the default option. in any case i will write more tests and i still need to fix the unicode literal repring

should i be using `pprint_thing` anytime (even a trivial string) i'm formatting a string?

`pprint_thing` is actually all about doing unicode properly. whenever you have something
that goes to the console, it's best to use pprint_thing. using it also makes sure
things are kept as unicode until it's time to encode them into a specific encoding.

well that's just dandy! thanks

@y-p @jreback u ok with this?

what does this fix?

just tacks on the last element of a sequence when the user has set max_seq_items to something > 0

ok as long as @y-p ok

related #3456, #3391 and especially https://github.com/pydata/pandas/issues/3426,
would have preferred matching behaviour to numpy: [Foo_N ... Foo_N]

I can do that. No problem. 

@y-p N is the `display.max_seq_items` option value? should it be N / 2?

no preference, if possible align with numpy, I believe it has an equiv in set_printoptions.

closing in favor of #3974.
BUG: fix python3/2 numpy import_array macro build issue with clang
State: closed

closes #3872. no really, it does.


Labels:

Comments:
famous last words :)

i wish u could test different compilers on travis...

@rhstanton can u test this? or anyone with a fairly recent mac? i don't remember when they switched to clang

Yup. Works for me using Anaconda on a very recent Mac.

From: Phillip Cloud <notifications@github.com<mailto:notifications@github.com>>
Reply-To: pydata/pandas <reply@reply.github.com<mailto:reply@reply.github.com>>
Date: Monday, June 17, 2013 1:02 PM
To: pydata/pandas <pandas@noreply.github.com<mailto:pandas@noreply.github.com>>
Cc: Richard Stanton <stanton@haas.berkeley.edu<mailto:stanton@haas.berkeley.edu>>
Subject: Re: [pandas] BUG: fix python3/2 numpy import_array macro build issue with clang (#3936)

@rhstantonhttps://github.com/rhstanton can u test this? or anyone with a fairly recent mac? i don't remember when they switched to clang

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3936#issuecomment-19570659.

https://travis-ci.org/pydata/pandas/jobs/8174283

this is starting to get annoying...

To annoy you even more, the latest version (as of 5 minutes ago) actually now compiles fine on my Mac. No idea why...

From: Phillip Cloud <notifications@github.com<mailto:notifications@github.com>>
Reply-To: pydata/pandas <reply@reply.github.com<mailto:reply@reply.github.com>>
Date: Monday, June 17, 2013 2:33 PM
To: pydata/pandas <pandas@noreply.github.com<mailto:pandas@noreply.github.com>>
Cc: Richard Stanton <stanton@haas.berkeley.edu<mailto:stanton@haas.berkeley.edu>>
Subject: Re: [pandas] BUG: fix python3/2 numpy import_array macro build issue with clang (#3936)

https://travis-ci.org/pydata/pandas/jobs/8174283

this is starting to get annoying...

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3936#issuecomment-19576454.

that's cuz i just merged the fix :) the build didn't pass travis tho bc of these google network error tests. i keep getting these intermittently and it's annoying. not sure why tho i think @gliptak is working on it.

@cpcloud please see #3893
FIX: StataReader
State: closed

Fix for a bug in StataReader resulting in errors when reading Stata file with string columns


Labels:

Comments:
can you add a test case for this?

e.g. write a df with multi-dtypes (including object), and read back in to compare?

Thank you for the suggestion - I was already thinking about how I can create a proper unit test, since I couldn't just use the real-world dataset where the issue happened first.

I have adapted the commit, now also containing a fix for StataWriter that was necessary to make the unit tests work.

as an aside (and this could easily be put in 0.12)

I think it makes sense to support `index_col` in `read_stata`, and `as_index=False` in `to_stata`, though I am not sure if you can store any meta-data in the `dta` file? (e.g. to indicate this column)?

also....I am not sure how much support stata has for dates, be sure to convert to/from `datetime64[ns]` let me know if you have any issues with this

keep in mind that `datetime64[ns]` CAN have `NaT` in there (equivalent of `nan`). You are of course free to raise if you need to if you encounter these (if stats cannot support), again I don't know the limitations

Updated. Now also testing datetime.

this ready to go? (the failure is unrelated to you)

I think this is ready.

great will merge in a bit

thanks again!
PTF no more
State: closed

The travis network build cache (activated via commit message, for whitelisted commiters) was added a few months back. It worked well for a while, but ultimately 
it's seen little use in practive by commiters. 

While it was gratifying to see this actually worked, all the additional
magic in ci/ doesn't pay for itself in practice, so I've decided to retire this
unless there are strong objections and get a simpler ci/ in return.

The build cache system got an upgrade as a side-effect of this 
work and `scripts/use_build_cache.py` remains a useful bit of kit.
Not entirely wasted effort after all.

xref https://github.com/pydata/pandas/pull/3383


Labels:

Comments:
Thanks for all the memories.
TST: Fix error in assert_produces_warning.
State: closed

When I copied this over and sent to @cpcloud,
I thought the indentation was wrong and changed it.
Turned out I was wrong -- sorry about that!

Now all the doctests pass...


Labels:

Comments:
@jtratner which doc tests are u referring to?

the doctests in assert_produces_warning. run `python -m doctest -v
pandas/util/testing.py` and you'll see what I mean.

On Sun, Jun 16, 2013 at 7:33 PM, Phillip Cloud notifications@github.comwrote:

> @jtratner https://github.com/jtratner which doc tests are u referring
> to?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3927#issuecomment-19521197
> .

k this is good 2 go...any objections? @jreback ?

ok

cool so @jtratner rebase and i will merge

@cpcloud I think this is actually up to date.

thanks
CLN: Replace bare exceptions with more descriptive ones
State: closed

This takes care of the low-hanging fruit (really obvious Exception changes), there are still a bunch more to change, but I thought I would split it up into parts and deal with external exceptions before messing with anything that might potentially be raised internally. [I rebased this slightly to remove a few comments - but it did pass previously so it should be fine]


Labels:

Comments:
here's the hard part
need to also change the corresponding test (if there is one)
to the new exception

you can easily do this by temporarily not raising (comment it out)
then the test should break and u can fix it

if it doesn't break then we r not testing it

tests are almost certainly for bare Exception

@jreback okay, will do :)

@jreback I've been writing out test cases for all of this, but it's involving some minor changes all around too. Would it be okay if I split this up into multiple PRs? (for example, right now I have test cases edited for stats, tools, tseries and sparse, could I submit those as PR and then continue working?)

I'm just concerned that it's going to get more and more difficult to merge the more files that are changed.

yes smaller PRs prob easier in this case
BUG: Add import for 'is_datetime64_dtype'
State: closed

This import was missing from `core/groupby.py`. This also suggests that there is an issue with the test cases, given that no test triggers this issue (suggesting that there is no current test that triggers the `elif` condition below).

``` python
    def _try_cast(self, result, obj):
        """ try to cast the result to our obj original type,
        we may have roundtripped thru object in the mean-time """
        try:
            if obj.ndim > 1:
                dtype = obj.values.dtype
            else:
                dtype = obj.dtype

            if _is_numeric_dtype(dtype):

                # need to respect a non-number here (e.g. Decimal)
                if len(result) and issubclass(type(result[0]),(np.number,float,int)):
                    result = _possibly_downcast_to_dtype(result, dtype)

            elif issubclass(dtype.type, np.datetime64):
                if is_datetime64_dtype(obj.dtype):
                    result = result.astype(obj.dtype)
```


Labels:

Comments:
u could run coverage and see if it agrees with your suggestion

@cpcloud Not necessary - the missing import is on master, if all the tests pass with the missing import, the line can't ever have been executed during the tests.

ah yes indeed

Looking at this now 

this can only be hit if the datetimes are not views and not datetime64[ns]
sometime which is not possible
I don't recall what happens to dt during groupby exactly ( whether they r converted with a view or not)

Do you want to remove the line altogether?

On Sun, Jun 16, 2013 at 6:54 AM, jreback notifications@github.com wrote:

> Looking at this now
> 
> this can only be hit if the datetimes are not views and not datetime64[ns]
> sometime which is not possible
> I don't recall what happens to dt during groupby exactly ( whether they r
> converted with a view or not)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3920#issuecomment-19510774
> .

I am not sure it is ever hit (as u have noticed)
let me see if I can figure out what it's supposed to do

@jreback any luck here?

tomorrow

this was being called incorrectly (and uncessarily), then trapped by try: except, all fixed up; closing this PR (fixed in #3934)...thanks for finding this!