closes #39 batch processing now processes links as they are found