BrandonFors/pandas-github-issues huggingface dataset:
TST: Changed the error raised by no tables in data.Options
State: closed

Tests were failing if the scraper got the webpage but there weren't any tables in it. Added condition to raise RemoteDataError if no tables were found.  Still IndexError if it can't find the correct table.

fixes #7335


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Is it ever the case that there are more than 0 tables found but a table location is invalid?

If 2 tables are found, and you wanted location 3

Not really a `RemoteDataError` in that case though. Can you put a conditional that will only raise a `RemoteDataError` if 0 tables are found

Good point.  I was trying to think of how to make fail in the case that Yahoo changes their tables around.

@jreback comments? i think this is ok

ok

@dstephens99 thanks!
BUG: ix should return a Series for duplicate indices
State: closed

closes #7150

it seems like there's a better way to do this .... 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
let me have a look

@jreback any thoughts on this?

I will take a look tomorrow

ok

closing in favor of #7402
BLD/TST/DEV: update tox to use python 3.4
State: closed

- now installs any optional dep not relying on numpy for better local test coverage
- you can now pass arguments that will go to nosetests (e.g., for running network tests)
- removed tox_prll.ini because you can `pip install detox` and it will automatically build run and test across the configuration in tox.ini


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jreback you ok with this? just a bit of pruning ...

seems ok
not that I use tox 

ok ... it's useful for not having 5 different virtualenvs when you want to test
BUG: correct Period comparisons
State: closed

closes #7376


Labels:
name: Bug

Comments:
@jreback @jseabold comments? passes except a random failure on 2.7

looks ok

I'm not sure I really understand the implications of the change, but looks harmless enough?

Should probably go in the release notes if for some reason someone was relying on the TypeError being raised. The rest of the magic methods still raise a TypeError, though I doubt it's worth worrying about consistency here? I can't see much of a reason for it.

yep there will be a doc blurb in v0.14.1.txt
BUG: resample raises ValueError when NaT is included
State: closed

Closes #7227. `resample(how=count)` should work now.

There are some aggregations which doesn't work with `TimeGrouper` yet, I'll open separate issue. 


Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
There seems to be 3 issues related to aggregation.
- var/std/mean raises `ValueError` when group key contains `NaT`. (Error will be changed after this PR, but still fails)
- `size` raises `AttributeError` regardless of `NaT` existence. 
- first/last/nth looks work, but `BinGrouper` seems to sort rows differently from normal groupby. Thus, result of normal groupby and `TimeGrouper` can differ. If we test (like `test_aggregate_normal` included in this PR), it fails.

The PR requires `Index.insert` to preserve index name (#7386). I'll rebase once it merged.

why does `size` raise? (this should work)

how is the `BinGrouper` sorting different? can you show an example?

Issued #7453 to explain `size` and `first/last/nth` issue, because these are irrelevant from this issue.

ok....going to merge this, and use #7453 to address the other issues
BUG: infer_freq results in None for hourly freq with timezone
State: closed

Found fix #7318 was incorrect. It works for lower frequencies than dates, but not for higher freq than hours.
 I couldn't detect as test cases are also incorrect...

This should work for all the freqs with timezones.

NOTE: I couldn't use normal `tslib.tz_convert` which returns incorrect results maybe caused by DST. Applying `tslib.tz_convert_single` can work though.


Labels:
name: Bug
name: Frequency
description: DateOffsets
name: Timezones
description: Timezone data dtype

Comments:
thanks

hmm, this broke in 3.4: https://travis-ci.org/pydata/pandas/jobs/26939770

this is broken in 2.7 as well, just bisected it

@sinhrks going to revert this

@cpcloud also putting 3.4 on the main build (and 3.2 on experimental, just to have it run)

cool, just reverted d91d145
BUG/REG: fix float64index -> mixed float assignment
State: closed

closes #7366


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
i see ... i think travis is making a "fake" merge commit to test PRs against master but runs the actual unmerged version on the submitter's branch
BUG/REG: fix float64index -> mixed float assignment
State: closed

closes #7366


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
WTF? I pushed fb38fb6 and it says merged here: https://github.com/pydata/pandas/commit/3ea8c7ad6c0032d87aca5c82e0efba34d6cb9034 from the link on travis

https://travis-ci.org/pydata/pandas/jobs/26909698

Well, the fix did the trick. I tested from your repo.

glad it works for you! just a little confused why travis has a commit that doesn't exist on my end but somehow exists in the GitHub ether and travis is able to check it out and run it. somehow my test is there but not the fix :confused: 

that weird Travis bug

it somehow picks up a different merged commit

good ol' rehash to the rescue :sunglasses: 

well for one thing this checkout line is wrong:

`git fetch origin +refs/pull/7367/merge` as there was only ever a single commit without a merge yet somehow one was created on GitHub

shoudl be ok now ... it's checking out directly from the hash

ok i'm going to get out of the car and get back in .... make a new pr because something is really screwy .. commit hash is there but the changes are not
TST7337: Try to resolve windows test failures.
State: closed

This change fixes broken tests on windows following `ENH: Support dateutil timezones`

closes #7337 


Labels:

Comments:
Closing as change being merged already.

ha....didn't merge yet....is this exactly the same?

yes, it's just that single commit.

merged via https://github.com/pydata/pandas/commit/662944666f770e537dfc4a71bb0c1a2c5e89140c
PERF: better dtype inference for perf gains (GH7332)
State: closed

closes #7332

for some reason `int64` was not being looked up correctly in the _TYPE_MAP table.
changing to work on name/kind makes this more efficient.

separately better inference of timedelta conversions from i8 were not being done
so that yields a big boost

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
dtype_infer_timedelta64_2                    |   3.9814 | 1765.3640 |   0.0023 |
dtype_infer_datetime64                       |   4.8450 | 1825.3207 |   0.0027 |
dtype_infer_int64                            |   0.7904 |  18.5820 |   0.0425 |
dtype_infer_float64                          |   0.8090 |   1.0493 |   0.7710 |
dtype_infer_float32                          |   0.3430 |   0.3823 |   0.8971 |
dtype_infer_int32                            |   0.3897 |   0.4017 |   0.9701 |
dtype_infer_uint32                           |   0.5450 |   0.5583 |   0.9762 |
dtype_infer_timedelta64_1                    | 118.2277 | 117.6037 |   1.0053 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [e2f5a50] : PERF: use names/kinds for dtype inference on known types
Base   [87660ef] : Merge pull request #6968 from ahlmss/AHLMSS_0.13.1_ahl1
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
wow, quite a perf improvement!

FWIW it worked for me to fix this symptom on my 64-bit box; my 32-bit isn't accessible, but I'll give it a go later, just for sanity purposes.

gr8; I merged into master. lmk if any other issues. thanks for the report!
FIX: resample with fill_method and how #2073
State: closed

closes #2073


Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Resample
description: resample method

Comments:
@jreback ok with this?

ok

@hayd thanks

fyi, don't use `range` in test comparisons (fails on windows)

https://github.com/pydata/pandas/commit/c5c447892aa7041e3b25b2ce1eebae47519d590b

why is that?

because range is a generator, so its kind of like do `np.arange(..)` on a 32-bit platform gives you 32-bit ints (which then don't compare properly) of course on 64-bit it gives you 64-bits. lists otoh give 64-bit  (its only numpy arrays and generators which do this)

ok good 2 know!
add class=„pandas-empty“ to NaN-cells’ HTML
State: closed

This PR is related to #5330, but is much simpler.  There is no new API, this just allows styling empty cells using CSS, e.g. using

``` css
  td.pandas-empty { color: #eee }
```

to de-emphasize, or

``` css
  td.pandas-empty { background-color: yellow }
```

to let empty cells stand out.


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@cpcloud @TomAugspurger  ?

I should mention that I _do_ see a weakness in the condition (testing the formatted contents against self.fmt.na_rep).  However, currently the design seems to make it very hard to do better; I tried implementing this before and failed, and found this simple "solution" in the above-mentioned PR #5330.
Although it is not very clean, I personally feel that currently, it is an appropriate approximation (for testing the raw value against NaN).

@hmeine I see what you want to do here. I think we are looking for a more general soln (w/o too much refactoring / and no API breaking).Their are some ideas/suggestions in those linked threads. Just need someone to try an implementation.

I see, but I would say those are complementary goals.  Marking empty cells (and index/column headers, cf. #6488) would be totally non-invasive and scratch obvious itches.  The more general solution would be far more capable, but the above allows _today_ to add CSS styling e.g. in IPython notebooks (which I am using daily).

@y-p created an excellent base from which to start, in #5763 

check out this notebook (from that PR): http://nbviewer.ipython.org/github/pydata/pandas/blob/8c4f9571c08d5122f1d9e40c43dadab8b6f10bf5/pandas_HTML_styling.ipynb

in #3190 you can see all the linked issues and how many times people have requested little additions or wished pandas did some formatting thing slightly different. i totally get where you're coming from, but i think that most of the legwork was done by @y-p and we should take advantage of it rather than going for the needs of "today". if we make this change we have to support it for the foreseeable future. personally, i'd rather support an addition that is more general.

:+1: on using the soln @cpcloud just pointed, its way more general, not that hard, and almost done. IIRC just needs some tests. Would be willing to put this in once integrated on an experimental basis. Maybe under `core.html` namespace (so can add stylers in the future)

@cpcloud the actual PR is at #5763 

yep my bad thx for correcting

was gh autocomplete + enter :/

Ah, now I see that PR #5763 is actuallly much closer to what I proposed here than the template engine proposed in PR #3190.  Then, I understand why you would rather see the generic solution.  Unfortunately, I cannot help with testing ATM.

I think this PR can then be closed.
BUG: Series.map fails when keys are tuples of different lengths (#7333)
State: closed

closes #7333 seems to be due to the new behaviour of `Series`, which will automatically create a MultiIndex out of the dict's keys when creating a series to map the values. The MultiIndex doesn't match the values for the shorter tuples, so they fail to map.

The fix is fairly simple, just pass the dict keys explicitly as the index.

I've added a test case for this specific issue, but could add some more if others can see related issues that would arise from this. I'll try to finalize the PR (squashing into a single commit, rebasing if necessary) tomorrow if it looks good.


Labels:
name: Bug

Comments:
this seems like a bug in the `Series` constructor, i.e., passing tupled-keyed dicts should do the right thing

probably related to the change to use multiindex for tuple keys by default

hm nevermind ... i guess the constructor behavior makes sense from a mi point of view

pls add a release note to `doc/source/v0.14.1.txt` as well

Alright, I've added to the release notes and it seems like everything's good, so I've squashed into a single commit. Should be okay to merge if people are OK with it.

@cpcloud looks ok to you? (i'll clear conflict on merging)

Yep. 

@onesandzeroes thanks!
BUG: already mixed indexes should not sort
State: closed

cc @jreback


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
passes for me

Yeah but not passing on Travis :(

Some tuple edge case

@jreback can u take a look here? previously this was sorting (since tuples are sortable), then comparing against a `MultiIndex`. So this _is_ a breaking API change, albeit for non `MultiIndex` `tuple` `Index`es, which are rare.

the change is that sorting mixed dtype indexes doesn't work for _any_ objects, including things that _do_ have an ordering.

@jreback ok with this?

seems ok, I don't really understand what its supposed to do. doubt this is really an issue in practice (except the error of course!) so ok

`tuple`s have an ordering which means they can be sorted inplace no problem, but i check for `mixed` and `mixed-integer` `inferred_type` which tuples satisfy. i thought it was better to not sort anything that's mixed, than to special case tuples
BUG: fix read_sql delegation for queries without select statement (GH7324)
State: closed

Closes #7324.

The `if 'select' in ...'` check was introduced for mysql legacy compatibility, but introduced problem if the query did not contain a select statement (eg a stored procedure). This should solve this, and passes the test suite (locally).


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
you mean #7324, rigtht?

yep, changed

should add something to release? otherwise looks good!

yep, and maybe also a small test with a stored procedure

added release note and test. Only problem for now is that the test is not working (see failed travis). Strange thing is that I get a `Command out of sync` error when I call `read_sql("CALL proc")` (https://github.com/jorisvandenbossche/pandas/blob/sql-delegation/pandas/io/tests/test_sql.py#L1097) after creating the procedure with `engine.execute(proc)` (which resultproxy is automatically closed by sqlalchemy). And If I run the `read_sql` a second time, it is OK. @hayd @danielballan any idea? (I can just add a `try: .. except: pass` for the first call, but that seems not really correct)

My guess is something with transactions. I spent a few minutes today and couldn't get it. I could be wrong, but I thought I'd share that thought in case someone has more time to look at this before I revisit it.

@danielballan Thanks for the pointer, that solved it.

I tested the following approaches:

```
engine.execute(proc)
```

```
with engine.begin() as connection:
    connection.execute(proc)
```

```
connection = engine.connect()
trans = connection.begin()
try:
    r1 = connection.execute(proc)
    trans.commit()
except:
    trans.rollback()
    raise
```

And only the last did work, although I don't fully understand why, as I would think the other should be equivalent (http://docs.sqlalchemy.org/en/rel_0_8/core/connections.html#using-transactions), as sqlalchemy autocommits by default. But in any case, it works now (and it was only for the test, so not that important).
UNI/HTML/WIP: add encoding argument to read_html
State: closed

closes #7220


Labels:
name: Enhancement
name: Unicode
description: Unicode strings
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@klonuo would you like to try out this branch on your data and see if it's to your liking?

don't you have to: cc @klonuo ?

i did

@cpcloud I just tried it and it works fine

Thanks for your patience

great! thanks for the report. keep the issues coming . promise i won't bite anymore :)

@jreback going to merge this after whatsnew

yep
BUG: xlim on plots with shared axes (GH2960, GH3490)
State: closed

Fixes #2960
Fixes #3490. 

Summary of changes:
- for irregular timeseries plots, considers all lines already plotted when calculating xlim
- for plots with secondary_y axis, considers lines on both left and right axes when calculating xlim. There are tests for non-timeseries plots (always worked, but now a test to confirm this), irregular timeseries plots, and regular timeseries plots.

This does not fix #6608, which I now think is a fairly specific/rare case that could always be manually avoided by `x_compat=True`. I will still try and fix that, but it looks a bit uglier and I didn't want to hold back these more important changes.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@TomAugspurger ?

I see from another PR that @TomAugspurger is travelling... is there anyone else who could take a look at this? Seems like a nice patch to have in 0.14.1.

(admittedly I am travelling also)

@cpcloud can you take a look?

@rosnfeld pls rebase this

Rebased. Thanks for reviving this. Looks like @sinhrks is working on similar material in #7459 and #7515.

cc @sinhrks can you have a look?

Good catch. I had created test_secondary_y_irregular_ts_xlim to test a secondary_y case but as written it only makes sure that a "primary" (non-secondary) axis is taken into consideration when you plot on the secondary axis, but not the other way around. I've strengthened the test and updated the code.

@rosnfeld pls rebase

Rebased.

one more rebase pls (i just merged bunch of plot stuff)

Ok, rebased again.

ok, ping when green

Travis is green on my fork but will take a while on the main pydata one I think. I am in Asia at the moment and unfortunately need to sleep. :)

hahha...np...thanks

Alright, it's green now.

thanks!
BUG: Revisit Index.delete preserves freq
State: closed

Fix the issue that freq is preserved inappropriately, caused by #7302. Fix preserves freq if edge element is being deleted (same as #7299). 
CC: @rosnfeld


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
ok looks good thanks

I tried the same exercise as before, and it didn't work for me:

```
In [1]: ts = pd.Series(1, index=pd.date_range('2000-01-01', periods=10, freq='D'))

In [2]: ts[:5].index
Out[2]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-01, ..., 2000-01-05]
Length: 5, Freq: D, Timezone: None

In [3]: ts[[1,2,3]].index
Out[3]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2000-01-02, ..., 2000-01-04]
Length: 3, Freq: None, Timezone: None

In [4]: ts.drop(ts.index[[5,6,7]]).index
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-3de5d002e589> in <module>()
----> 1 ts.drop(ts.index[[5,6,7]]).index

/home/andrew/git/pandas-rosnfeld/pandas/core/generic.py in drop(self, labels, axis, level, inplace, **kwargs)
   1462                 new_axis = axis.drop(labels, level=level)
   1463             else:
-> 1464                 new_axis = axis.drop(labels)
   1465             dropped = self.reindex(**{axis_name: new_axis})
   1466             try:

/home/andrew/git/pandas-rosnfeld/pandas/core/index.py in drop(self, labels)
   1801         if mask.any():
   1802             raise ValueError('labels %s not contained in axis' % labels[mask])
-> 1803         return self.delete(indexer)
   1804 
   1805 

/home/andrew/git/pandas-rosnfeld/pandas/tseries/index.py in delete(self, loc)
   1618         """
   1619         new_dates = np.delete(self.asi8, loc)
-> 1620         if loc in (0, -len(self), -1, len(self) - 1):
   1621             freq = self.freq
   1622         else:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

Thanks for your confirmation. I didn't imagine `loc` can be list or array.. I'll reconsider the logic.

@sinhrks is this updated with @rosnfeld example?

@jreback @rosnfeld Updated to accept `int`, `slice` and `np.array` as the same as `np.delete`, and added test cases for these.

@rosnfeld can you give this a try?

Looks good.

thanks @sinhrks !
BUG: single group series should preserve group name
State: closed

Extract currently doesn't preserve group names when a series is returned. This
PR fixes that.


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
@jreback good 2 go after a doc blurb?

sure

what's the priority of travis jobs it seems like i have to wait for a long time sometimes and then other times i don't wait that much. really frustrating 

this passed a while ago and it seems like it'll never run on master

https://travis-ci.org/cpcloud/pandas/builds/26600221

merging
BUG: quantile ignores axis kwarg
State: closed

Closes https://github.com/pydata/pandas/issues/7306

We may want to wait on this till later today while I figure out what will happen in https://github.com/pydata/pandas/issues/7308


Labels:
name: Bug
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
This should be ok once Travis passes.

One odd thing is that `numeric_only` is _always_ considered by column, even when `axis=1`.
So if you have something like

``` python
[[1,    2,  3],
 ['a', 'b', 4]]
```

It will exclude the first two columns since they aren't numeric.  But this is the same as the previous behavior.

Just to expand on that last comment. As before, we get:

``` python
In [34]: df
Out[34]: 
   0  1  2
0  1  2  3
1  a  b  4

In [35]: df.quantile(.5, axis=1)
Out[35]: 
0    3
1    4
dtype: float64
```

So the first two columns are excluded since they're non-numeric, so we're doing quantile on just the last column:

``` python
In [38]: df[2]
Out[38]: 
0    3
1    4
Name: 2, dtype: int64
```

Since `axis=1`, we take the median of each length 1 array and get the same thing back.

What I think should happen is we transpose _first_, and see that the first col is numeric and calculate the quantile on that column:

```
In [52]: df.T
Out[52]: 
   0  1
0  1  a
1  2  b
2  3  4
```

So `df.quantile(.5, axis=1)` should be equivalent to `df.T.quantlie(.5, axis=0)`

But this would potentially break existing code so I'll hold off on that till the future.

I added one more test covering this; I'll merge when that's green.

gr8 looks good

thanks @TomAugspurger
BUG/TST: should skip openpyxl before testing version compat
State: closed


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite

Comments:
looks ok (make that it skips correctly on numpy 1.9_devv build)

@jreback this passing all builds on my branch ok 2 merge?

https://travis-ci.org/cpcloud/pandas/builds/26584095

ok