iParadigms/walker — pull request #121 by pchristopher1275 (2014-12-26 19:28:24+00:00)
Title: TRN-414 complete patch with tests passing.

Ticket found at https://jira2.iparadigms.com/browse/TRN-414

The following list of links/domains each have some kind of endless repeating behavior that makes them useless for us to crawl. Thus these domains are disabled; we only really want to crawl them if we're not going to get junk. These can also be found at https://confluence.iparadigms.com/display/SEU/Walker+Project+Plan
For each one:
Explore whether it's the target site's problem or how we are parsing their links
If we can parse better so that we will no longer have an issue on the site, implement the fix; due to link filtration some of these may no longer be a problem, but at least write a test to verify that
Create a separate story for all "fixed" sites; we'll need to reenable them but in most cases we will probably also want to clear out the bad data too (and remove them from the wiki page)
For sites we cannot fix, update the exclude_reason on those domains to explain why they are excluded
....