ecprice/newsdiffs — pull request #5 by payload (2012-08-26 22:21:20+00:00)
Title: Extensibility

In this branch I move the *Article classes into a `website/scrapers/` directory. Each in one file. Alongside the domain and url to fetch from. I also split `NytArticle` off of the base `Article` class.

`scraper.py` has now another option `--fetch`, which I used to quickly test different scrapers without to run all of them. It matches them vaguely enough such that I can say `--fetch cnn` or `--fetch "tagess bbc" --fetch cnn`.

To load `Article` classes all modules in the scrapers directory are loaded and their content is filtered for `isinstance(article, scrapers.Article)`. The mapping between domain and classes and the urls to fetch from are filled with the data written in each `Article` class, filtered by the arguments to `--fetch`. Works :-)

Something which is in this branch but does not belong here: the printed output is a little bit more verbose for `update_articles` and more "oneliner oriented" for `update_versions`. Hope you like it.

I also moved the utility functions and the monkey patches in `scraper.py` to their own files next to it, so the file is not so crowded with … stuff. In which folder they really belong, I don't know. Maybe right there where they are. There are some wildcard imports, which need to be looked at. As I moved code I also noticed (again) that putting imports near the places where they are needed, is faster to handle. The usual downside is, that missing module dependencies are only noticed late at runtime… well.

I didn't look into the web stuff to add scraper domains there too.

So TODO for this branch is still:
* looking at wildcard imports and change them to something less hurting
* where to put `_monkeypatch.py` and `_utils.py`
* configurable, more automatic web frontend stuff, when a scraper is added/removed
* say what you think :-)