TST: Change test_html to use stored data + mark other
State: closed

Changed double decorated tests previously `network` and `slow` so they actually get run + stored usda_data such that the tests don't fail on network access.


Labels:

Comments:
thanks must have missed that when i wrote them. i think slow should be removed since either slow or network tests get run on travis iirc

also there's already `spam.html` in `pandas/io/tests/data`. i explicitly want to connect to the network here just in case something has changed, although there's no reason to believe the composition of spam will change anytime soon. so probably good to use `spam.html`

Okay, I'll make that change. Do you want to store data locally for these or leave with the network request? html file is 32K

the banklist data set for example is updated every monday and so it's basically frozen in time and you cannot compare the data obtained with that that is stored locally. i get 28k (for spam) but that's splitting hairs... probably ok to leave local

i did `du -h $(find . -name 'spam.html')`

Huh? Currently it's not local for the html file test. (most of the other test cases are). Plus these are basically just checks that lxml and BeautifulSoup actually work as expected.

Huh? Currently it's not local for the html file test. (most of the other test cases are). Plus these are basically just checks that lxml and BeautifulSoup actually work as expected.

it's ok to change to local if u want that's what i meant

Sorry for the double comment there - poor connection. The html file that the lxml and bs4 tests check for is 32k.

Anyways I don't have a strong opinion. I'll edit this to remove slow soon. 

oh u mean the one u added? or the one that is received by `urlopen`? i'm confused....all i'm saying is that if you want to change to use the local files instead of calling the network you can...they are there, spam is called `spam.html` and the banklist data set is called `banklist.html`...make sense?

@cpcloud yeah, that does. just wondering if you think it's worth it - sorry I wasn't clear earlier

Since these are essentially smoke tests for the backend you're right best to move to slow decorators and remove the network decorators. 

one big reason why these take so long is that they are processing a huge
file. Is that necessary or can I reduce it to just the relevant portion
(i.e., `<htm><head></head><body><table>TABLESTUFFHERE</table></body>`)?

On Mon, Jun 24, 2013 at 9:32 AM, Phillip Cloud notifications@github.comwrote:

> Since these are essentially smoke tests for the backend you're right best
> to move to slow decorators and remove the network decorators.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4009#issuecomment-19906757
> .

wanted to test on possibly invalid html from the wild. if you can find a smaller file that works that's great too

@cpcloud hmm...well, just going to leave it as slow I guess. Sort of a bummer that it can't be both.

if you come across another data set in the wild please put up a PR, would love to have both just don't have time to check right now for data sets from pages that are both small and rife with HTML errors. [this scraping framework](http://doc.scrapy.org/en/latest/intro/tutorial.html) might be of use for finding all kinds of different HTML tables.

@cpcloud Changed double decorated tests previously `network` and `slow` so they actually get run + stored usda_data such that the tests don't fail on network access. The html file is 32k, not sure whether that's too large.

i'm confused..why not just use the spam data set that's already there?

@cpcloud ohh, good point. Somehow I thought there was some decision to use a different file.

changed it.

@jreback look ok? i think this can be merged

looks ok

Does this need rebase?

@cpcloud more fun with network failures! :D https://travis-ci.org/jtratner/pandas/jobs/8559725 [pretty sure this is off of master and not due to this commit]

darn. that is off of master. i'm not even sure what the issue is there. i'll delve deeper. i really wish i had a way to reproduce this stuff. i may end up just setting up my own CI on my machines at home

@cpcloud Maybe you could set  up something that tracks all connections open at any one time (maybe mod urlopen to do it?) then, when it raises an error, have it print the entire set of open network connections.

[there is no hope](http://stackoverflow.com/questions/8734617/python-django-badstatusline-error). will just catch this exception...teh interwebz r tricky

@cpcloud that seems like the best bet here. Given the confusing-ness of the error, definitely something that the datareaders should handle themselves.

badstatusline exceptions are just like throwing `Exception`: completely uninformative if you want to use it to handle things gracefully