mozilla/socorro — pull request #600 by peterbe (2012-05-18 22:09:56+00:00)
Title: Bug755535 ftpscraper cronapp2

lars, rob: r?

A couple of things to focus on:

* Instead of doing `import lxml; ... lxml.html.something` I opt to do `import lxml.html; ... lxml.html.something`
This is the only way I can make it work consistently on OSX with 2.3.4 and 2.3.3. Damn binaries!

* The `except urllib2.URLError` is still in there. Maybe we should remove these and instead rely on backfilling?? Note that `buildlib` is sending `connection.commit()` which would mean it would be hard to put this whole job in a single transaction.

* I opt to NOT delete the old ftpscraper, it's tests and it's bash script. Depending on feedback I can add a commit that removes that or we can wait. Thoughts? On that note, should I then also add this job to the crontabber.ini in configs.

* Check out `getNightly()` it used to have a `return` statement in there. Was that really correct? It would effectively stop the generator instead of skipping those that are NOT 'en-US' or 'multi'.

Test coverage is 95% and spans both integration and unit tests.