clips/pattern — pull request #24 by relwell (2012-12-11 23:43:50+00:00)
Title: Generator for MediaWiki/Wikia

This changeset introduces MediaWikiArticleSet and WikiaArticleSet classes. They are generators that provide instances of MediaWikiArticle.

MediaWikiArticleSet iterates using MediaWiki.list(), which uses the traditional MediaWiki API to list all pages, and then the MediaWiki.search() command to access those pages based on their title.

WikiaArticleSet uses MediaWiki.list() to get the article IDs, and then uses those IDs on Wikia's search backend to access the source for multiple pages at once.

This is still all considerably slow, and likely very hard to scale for really data-intensive stuff, but it's at least a step in the right direction.