Added a new interface for storing links. This allows for numerous link store backends (e.g. not just in memory slices) and gives flexibility over the order of providing links to the spider.