BrandonFors/pandas-github-issues huggingface dataset:
BUG: hidden ticklabels with sharex and secondary
State: closed

Closes https://github.com/pydata/pandas/issues/9158

``` python
import pandas as pd
d = {'A' : [1., 2., 3., 4.], 'B' : [4., 3., 2., 1.], 'C': [5, 1, 3, 4]}
df = pd.DataFrame(d, index=pd.date_range('2014 10 11', '2014 10 14'))

axes = df[['A','B']].plot(subplots=True)
df['C'].plot(ax=axes[0], secondary_y=True)   # x tick labels dissappear when this line is executed
```

![gh](https://cloud.githubusercontent.com/assets/1312546/7216419/ffe3bc2e-e5c4-11e4-89de-d18e5e59d3b0.png)

Introduced in the subplot refactoring in https://github.com/pydata/pandas/pull/7457

cc @sinhrks

The fix is pretty simple, but I'm worried I may have broken other things. No tests failed though, so hopefully we're good.


Labels:
name: Visualization
description: plotting

Comments:
My reformatting hides the change.

The only difference is in the call to `_handle_shared_axes`, `naxes` is now `nrows * ncols`, instead of `len(all_axes)`. The problem was that `all_axes` is 3 now, since there's the original two subplots and the additional 3rd plot, which is getting plotted on the first subplot. This confused `_handle_shared_axes`.

looks good. minor doc issue.

@TomAugspurger looks ok, pls rebase.

needs a rebase

Rebased... will merged later today.

The failure [here](https://travis-ci.org/pydata/pandas/jobs/59048983) was a build failure on a job that doesn't run the plotting code. Going to merge

Well, I messed that up. Pushed before `--continuing` my rebase, and it looks like github has it's done with this PR

this problem seems to be still existing in pandas 0.17.0, on subplots with at least two rows:

``` python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

mat = np.random.rand(10,2)
df = pd.DataFrame(mat, columns=['A','B'])

fig, axes = plt.subplots(nrows=2, ncols=2)                                 
df.plot(secondary_y='B', ax=axes[0, 1], legend=False)
```

![image](https://cloud.githubusercontent.com/assets/5053963/10861467/dd738f4e-7f56-11e5-8db9-e30582765b8f.png)
PERF: Utilize mixed dtypes in df.count() with MultiIndexes
State: closed

@jreback Same underlying cause as #9136 but the solution is a bit more involved here.

Basically, we're transposing a potentially mixed-type frame before calling `notnull(frame.values)`; the same result can be obtained by deferring the transpose until after `notnull` gives us a non-mixed frame.

Here are the vbench results:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_count_level_axis1_mixed_dtypes_multi   |  82.2484 | 1489.9830 |   0.0552 |
frame_count_level_axis0_mixed_dtypes_multi   | 101.2537 | 1737.6347 |   0.0583 |
frame_count_level_axis0_multi                |  51.0643 |  51.3713 |   0.9940 |
frame_count_level_axis1_multi                |  98.5887 |  82.6767 |   1.1925 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [04ec4c6] : PERF: Utilize mixed dtypes in df.count() with MultiIndexes
Base   [def58c9] : Merge pull request #9128 from hsperr/expanduser
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
minor change. looks good otherwise.

@jreback Fixed and Travis is passing.

merged via https://github.com/pydata/pandas/commit/9d4f423184e6ae86762f9b4d553e7b9612630177

thanks!

Awesome, thanks.

no thank you!
Check whether GBQ Job is finished
State: closed

xref #8728 

jobComplete can be False in query_reply. Simply checking for the existence of the field isn't enough, as a False value will cause a KeyError when checking for query_reply['totalRows']


Labels:
name: Bug
name: Duplicate Report
description: Duplicate issue or pull request

Comments:
@shoyer I like the get method. Added a comment to make it blatantly obvious why the loop is there.

@jacobschaer pls have a look

Isn't this similar to https://github.com/pydata/pandas/pull/8728 ?  We are waiting for Google's response to http://stackoverflow.com/questions/27810116/google-bigquery-request-too-large

Looks like this is the same issue as #8728. The code is exactly the same. I can see in that issue that the existing code came right from Google's example, however the [API reference for getQueryResults](https://cloud.google.com/bigquery/docs/reference/v2/jobs/getQueryResults)  regarding the jobComplete key states:

> Whether the query has completed or not. If rows or totalRows are present, this will always be true. If this is false, totalRows will not be available.

So clearly we need to check the value of jobComplete before moving on to grabbing totalRows. The thread you've referenced at SO is regarding an issue with inserts, why is that holding up this unrelated change?

It's being held up because we wanted to get the intergration suite to pass before giving this the go ahead. The other PR I referenced had a couple of changes from @shaun-schaefer that was supposed to go in as well and needed testing. At this point though, I am wondering if we should just proceed with the understanding that the test suite has a known issue.

Gotcha. I'd argue that having read_gbq() return data reliably is more valuable than passing all integration tests, especially since the BQ API is still evolving.

@jacobschaer how's the testing coming?

closing in favor of #8728 (its the older issue).
TST: Fix failing test due to revision in WB data #9115
State: closed

Closes #9115  

I just round the expected and result data for the failing test.  I have no idea why the WB would be revising 2003 and 2004 data, but they did.

I would have redesigned the test, but spending that time has minimal benefit.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
looks ok. ping on green. thanks!

ping?

thanks!
ENH: Expanduser in to_file methods GH9066
State: closed

closes #9066

First time contributor, tried to follow all rules as good as possible.
If something is missing please let me know.

As specified in task 9066 tilde in a pathname get resolved into a absolute path.
In case the object is a buffer the buffer will be returned.

I did not add additional tests, since I could not find any module explicitly testing pandas/io/common.py
if that is necessary please let me know how to do that. The current tests are passing on my system and seem to cover the method below.

Best Regards,


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jorisvandenbossche worth adding a tests_common.py in pandas/io ?

I don't know if this is easy to test (as the home dir depends on where the tests are run, of course can reuse `expanduser` to just test that the feature is there), but if so, certainly a good idea to add a `io/tests/test_common.py`

I now use isinstance instead of try/except to see whether the path might be expandable. I also tried rebasing the new version so it will only be two commits again but maybe my git knowledge is not good enough to do so, if that is necessary please let me know and I will read up/figure it out.

about testing, not sure if its good practice to test against builtins but we can test whether get_filepath_or_buffer() returns the same as os.path.expanduser given a string input and returns the StringIO given a non string input/url.

I will add some tests for that, thanks.

That is what I meant with "I don't know how easy it is to test", as it is indeed not very useful to test builtin behaviour. But what we _can_ test, it just our API, the fact that the feature is there and will not accidentally be removed later on. So in that case, I think it is not bad to just reuse `expanduser`, or maybe just assure that something is added to the string)

@hsperr This looks great! Could you please squash this into a single change? See [this page](https://github.com/pydata/pandas/wiki/Git-Workflows#git-rebase) if you're not quite sure what that means.

@shoyer Thanks a lot for the link to the rebase page, I indeed did not feel super comfortable with that but I think I managed to squash/fixup everything into a single commit. I also took your version for the changelog, thanks! If there is anything more let me know and hope I can contribute more in the future! 

Thanks @hsperr !
PERF: use labels to find duplicates in multi-index
State: closed

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
multiindex_duplicated                        | 139.2883 | 469.6666 |   0.2966 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [2b99b80] : use labels to identify duplicates in multi-index
Base   [0fe43a6] : Merge pull request #9120 from minrk/nbviewer-link

fix an nbviewer link in tutorials
```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
this on top of  #9061 ?

[`get_flat_ids`](https://github.com/pydata/pandas/pull/9061/files#diff-720d374f1a709d0075a1f0a02445cd65R3514) is copy pasted from https://github.com/pydata/pandas/pull/9061. `MultiIndex.is_unique` has been re-factored.

ok, #9061 merged, so pls rebase this

pls add a release note as well

rebased and added release note. travis build all green.

@jreback actually i think it is better to have [`duplicated`](https://github.com/pydata/pandas/pull/9125/files#diff-584ad9f2559ff5ffb39ee605b3a20513R541) as a separate function as opposed to class method. so plz do not merge, and let me change the code.

@behzadnouri no prob. lmk

@jreback this is done!

merged via https://github.com/pydata/pandas/commit/8f4a3213ff352ce6e379ec654a53a0de5adb7d23 thanks!
Use tz.gettz() instead of zoneinfo.gettz()
State: closed

zoneinfo.gettz() seems to have problems (1 & 2) on system which do not install
the zoninfo tarball (e.g. Debian, Gentoo and Fedora) but rely on the system
zoneinfo files. This results in test failures (3 & 4)
tz.gettz() doesn't suffer from this problem.

1 https://github.com/dateutil/dateutil/issues/8
2 https://github.com/dateutil/dateutil/issues/11
closes https://github.com/pydata/pandas/issues/9059
closes https://github.com/pydata/pandas/issues/8639
closes #10121

Signed-off-by: Justin Lecher jlec@gentoo.org


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Windows
description: Windows OS
name: Timezones
description: Timezone data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
the is an api issue for dateutil

the current settings work on all platforms 
except yours

pandas should not be fixing the problem rather dateutil should

@jlec so this completely breaks windows. That is the reason I changed it in the first place. That said if you wanted to put in different imports (e.g. use the current code if on windows, else use what you did). I think that would be acceptable, until dateutil actually fixes this.

lmk

@jlec any interest in doing the change I propose above?

@jreback I didn't saw your direct reply. I will look into this this week.

@jreback which way do you prefer to check for windows, os.name or sys.platform?

use sys.platform != 'win32'

Sorry for the long delay. Somehow I don't get mails from GH correctly

@jlec can you update?

@jlec you need to rebase on master, see howtos in [here](http://pandas-docs.github.io/pandas-docs-travis/contributing.html)

It's not the wrong rebase, I have a merge commit. I will try to clean that up.

ok, pls add a release note in v0.17.0.txt mentioned all the above issues (you can just list them with a single note). and squash to a single commit.

Hopefully everything is now respecting the contribution guidelines. If not, please tell me what needs to be fixed.

ok, ping me when travis is green and I will merge.

@jlec thanks.
Remove bar line
State: closed

Closes https://github.com/pydata/pandas/issues/9088
Closes https://github.com/pydata/pandas/issues/7128

Votes on whether we should
1. Remove the line entirely
2. Add a parameter to `.plot` control whether the line should be added.

I vote for 1, but I can go either way.


Labels:
name: Visualization
description: plotting
name: API Design

Comments:
-1 in adding a parameter - this just looks like a bug
and could be done if the user really wants via matplotlib (the default is what u r doing as it seems more common)

Agreed, let's kill it. It's easy enough to add in manually afterwards.

minor doc issue. looks good.

@TomAugspurger I think you can merge this
API: Datetime-like indexes `summary` should output the same format
State: closed

Related to #6469. Made all datetime-like indexes `summary` should output the same format.
#### Before

```
didx = pd.date_range('2014-12-20', freq='M', periods=3)
pidx = pd.period_range('2014-12-20', freq='M', periods=3)
tdidx = pd.timedelta_range(start='1 days', freq='D', periods=3)

didx.summary()
# DatetimeIndex: 3 entries, 2014-12-31 00:00:00 to 2015-02-28 00:00:00
# Freq: M

pidx.summary()
# PeriodIndex: 3 entries, 2014-12 to 2015-02

tdidx.summary()
# TimedeltaIndex: 3 entries, '1 days' to '3 days'
# Freq: <Day>
```
#### After
- `DatetimeIndex` and `PeriodIndex` use `_format_func`. This made `DatetimeIndex` not to output hour/minutes.. when its frequency is more than a day.
- `PeriodIndex` outputs its `freq`
- `TimedeltaIndex` has no changes.

```
# DatetimeIndex: 3 entries, 2014-12-31 to 2015-02-28
# Freq: M

# PeriodIndex: 3 entries, 2014-12 to 2015-02
# Freq: M

# TimedeltaIndex: 3 entries, '1 days' to '3 days'
# Freq: <Day>
```


Labels:
name: Datetime
description: Datetime data dtype
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
somewhat separately the freq printing for TimeDeltaIndex is showing the actual obj and not the string repr

Yeah, `TimedeltaIndex.freqstr` is not compat with others. Can I fix this in the same PR, or shoudld be separate?

```
# DatetimeIndex
didx.freqstr
# M

# TimedeltaIndex
tdidx.freqstr
# <Day>
```

I think u can fix it here 
it shouldn't break anything (if it gets too complicated do a separate one)

@jreback, @jorisvandenbossche 
Updated, could you check?

Looks good to me!

thanks @sinhrks
TST: sql skip tests at class level if database is not available
State: closed

Added a `SetUpClass` to skip the full class when no connection can be made to the database, and made some of the other methods class methods, to be able to use them in `SetUpClass`

Eg if you are testing locally and don't have one of the databases available, then it will test the connection and skip each individual test, which can take some time.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
closes #7091 ?

@jreback no, it still something else I think. The issue is about the creation of the database itself, here it is just checking if a connection can be made for each class instead of for each test separately.
SQL: fix type conversion for Timestamp object typed columns (GH9085)
State: closed

Closes #9085


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@artemyk should only be a very small change

Maybe you should add `datetime64` to `_SQL_TYPES` also, so it works for legacy mode?

In fallback mode, `datetime64` is already converted to `datetime` (https://github.com/pydata/pandas/blob/v0.15.2/pandas/io/sql.py#L1351), so that should be ok 

Oh right, that makes sense!
Refs BUG-9099: Support simultaneous copy and dtype args in DataFrame init
State: closed

closes #9099
This is a somewhat brute-force fix for supporting copy and dtype arguments in DataFrame constructor. Some data might be copied twice if both arguments are set, but if I did the operations in a different order, one other unit test would fail.


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
perfect pls add a release note in v0.16.0 in bug fix section

OK, this time I actually read the contribution guidelines. ;-)

Added the info, new pull request:

https://github.com/pydata/pandas/pull/9105

ma.

On Thu Dec 18 2014 at 4:43:29 PM jreback notifications@github.com wrote:

> perfect pls add a release note in v0.16.0 in bug fix section
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9105#issuecomment-67494519.

@mairas thanks!
overflow bug in multi-index when checking for duplicates
State: closed

closes https://github.com/pydata/pandas/issues/9075


Labels:
name: Bug
name: MultiIndex

Comments:
@behzadnouri thanks!

the tests break on python 3.2 (we still support this?) due to the use of the `u'..'` syntax

See the extra builds below: https://travis-ci.org/pydata/pandas/builds/44414725

just need to do (technically we do support) - it's not hard to maintain 
though if others are dropping 3.2? 

u('value') instead of u'value'
Fixes .nth() with groupby multiple columns
State: closed

closes #8979

Let me know if this makes sense

Problem mentioned in issue 8979 too


Labels:
name: Bug
name: Groupby

Comments:
tests!

my coworker got a kick out of that :smile: 

test added (and method fixed, didn't copy it correctly)

looks good. pls add a release note as well (v0.16.0), bug fixes

Sorry, was out of the office for a bit. Pull request should be good now.

merged via https://github.com/pydata/pandas/commit/b3d1a059fd86577b46f080c3bd3661597530587f

thanks!
Closes #8943:  COMPAT: periods needs coercion to np.int64
State: closed

Let me know if this needs any work.

closes #8943 


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
does the test actually trigger the error?

pls add a release note (bug fixes in v0.16.0) as well

Yes it does.  In fact, it was overkill, so I simplified the test and set `periods` to just 10.  I added some comments inside the test to explain what I'm testing for and why it overflows.

I tested it on 32-bit Linux (in VirtualBox) without the type conversions and it fails as expected.

```
======================================================================
FAIL: test_time_overflow_for_32bit_machines (pandas.tests.test_index.TestDatetimeIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/garrettvb/Desktop/pandas/pandas/tests/test_index.py", line 1921, in test_time_overflow_for_32bit_machines
    self.assertEqual(len(idx1), periods)
AssertionError: 2 != 10
```

Also, I added "(bug fixes in v0.16.0)" to the commit message.  Is that what you meant by "add a release note"?

@Garrett-R see here for release notes: https://github.com/pydata/pandas/blob/master/doc/source/whatsnew/v0.16.0.txt

take a look at one of the older ones for examples.

Oh, haha, thanks!  I'll take a look.

Sorry about the typo.  OK, hopefully everything's in order now.

Thanks @Garrett-R !
BUG: Bug in MultiIndex.has_duplicates when having many levels causes an indexer overflow (GH9075)
State: closed

closes #9075 


Labels:
name: Bug
name: API Design
name: MultiIndex

Comments:
so this is implemented on a `MultiIndex` mainly for a perf benefit (e.g. it is order number of levels, rather than length of the index).

Should implement the trivial version of `Index` for compat. Essentially its `not self.is_unique`

this can overflow with int64 as well. it should follow something like [this](https://github.com/pydata/pandas/blame/e92df221aa958dc24a70cd42836f5b6f72d4b497/pandas/core/reshape.py#L485) and ideally factorized in one place

@behzadnouri you are probably right but I don't this is a practical overflow (unlike a groupby which deals with s theoretical space). can u come up with an example that actually does overflow? (obviously using int64)

[this commit](https://github.com/pydata/pandas/commit/c934e02da0a8f988c56310c2ca29fdb492c94c83) , which says:

> Fix int64 overflow issue when unstacking MultiIndex with many levels (#2616)

the [`get_group_index`](https://github.com/pydata/pandas/blob/e92df221aa958dc24a70cd42836f5b6f72d4b497/pandas/core/groupby.py#L3493) function is technically doing the same thing as in the first lines of `has_duplicates` function.

@behzadnouri 

what do you think?

closing in favor of #9101
ENH: add interval kwarg to get_data_yahoo issue #9071
State: closed

PR for the issue #9071


Labels:
name: Enhancement

Comments:
needs some tests for this

The Travis CI builds fail for an import error in an unrelated module. The PR pass all tests locally.

@jreback OK

you have to add tests to the PR

Added tests for 'monthly', 'dividend' and invalid intervals in test_get_data_interval.

@jreback I think the PR is ready for merge, the Travis CI failures "ERROR: Failure: ImportError (cannot import name json)" are unrelated to my change. All tests related to the pandas.io.data module pass on my machine with python2.7 and python3.4.

Is there anything more I can do?

pls add a release note (enhancements section, referencing the issue).

rebase on master and you should be good. ping on green.

Hopefully I did the rebase correctly.

I can't make sense of the Travis CI errors, some are unrelated to my changes and at least one appears to be a temporary failure of Yahoo! CSV API. I used the changes extensively a few days ago, so I'm quite confident of the code, but I experienced intermittent failures of the CSV API today.

IMO the PR can be merged as is. Thanks.

@alexamici well it appears to be failing on the routine you changed. 

@jreback short story: this is quite embarrassing, but the test function I wrote was total crap and I didn't notice as it was being skipped in my local testing. Now all should be well.

Long story: to my partial excuse I usually use py.test assert or doctests so I had no prior experience with assertEqual and friends, and I made a mess of the test function. Furthermore when running locally my tests were skipped due to lxml missing and I didn't figure it out until today. Last but not least, I was using the 'GOOG' symbol that has been redefined recently and queries for 2013 resulted in a Yahoo! API failure.

Tests are passing now! @jreback this is OK?

thanks @alexamici !

btw, if you would like to review the docs in remote.rst for any updates
(looking for a general review, not necessarily tied to this change).
TST: start tests for PeriodConverter
State: closed

@jreback The tests I started related to PR #9050


Labels:
name: Visualization
description: plotting
name: Period
description: Period data type

Comments:
https://github.com/pydata/pandas/commit/b69b9d3e208c0a0ac6e1f26ae1fb4ec10d0cd7bb (obviously you have more)

yep, I saw that. But I thought to add it both as a test for PeriodConverter (which then seemed to not yet exist), and as a plotting test (but only came around to the first one yesterday, and you already did the second). 

@jorisvandenbossche thought you had merged this, looks ok

there are two parts skipped for now in this PR:
- the datetime64 part is commented out -> there is already an issue about that: https://github.com/pydata/pandas/issues/9054
- the unicode test (`test_convert_accepts_unicode`) is failing, so I was skipping it for now (I copied the test from DatetimeConverter): @jreback this is something that should work?