rembo10/headphones — Issue #463: Request - Scraper Plugin Interface
Author: sbuser
Created: 2012-01-29 03:07:03+00:00
State: open

For quite awhile now I've been using external applications to scrape various sites for items of interest. These external apps then check a local library, if the item is missing the MB identifier is looked up and then passed on to Headphones.

There are several advantages to such a system. Most revolve around the way I prefer to collect - but there has also been a positive side effect. Since I'm adding by release id instead of artist I don't have thousands of update requests going to MB every day for full artist updates. In fact: for every release I seek there is typically 1 request to MB - ever.

In other words: one way to cut down on traffic to MB would be to avoid using MB as if it is a release news site. Rather: use the many hundreds of such sites that already exist for that and just use MB for the canonical metadata necessary to keep things well ordered.

A simple plugin interface would be very good for this. Each plugin would have a schedule and a cache of names->mbid relationships to avoid multiple lookups. Simple scraper plugins could be written by a wide array of people if you give them examples and make the interface robust.

In this way one could collect in a wide variety of ways (as opposed to just based on an artist) AND significantly reduce traffic to MB.

I hope this makes sense. I'm trying to avoid specifics about the scrapers since most websites that would be scraped would probably not be pleased about it - but you can see the potential, yes?