iParadigms/walker — pull request #120 by pchristopher1275 (2014-12-19 22:58:49+00:00)
Title: TRN-210 bug test implemented, and FetchResult.FetchTime set correctly.

Ticket content for https://jira2.iparadigms.com/browse/TRN-210 -->

See http://research-cr1.oak.iparadigms.com/links/banderabulletin.com
We cannot crawl that site because it disallows all unrecognized crawlers, so we expect a single fetch to have occurred showing the seeded link excluded due to robots.
However the main list shows it as Not Yet Crawled. Clicking on link history reveals that it was crawled once, but the fetch date is listed with the year 1754. This could be old data and an issue we've already fixed, but we should verify that even fetches that don't happen (i.e. due to robots.txt) get a correct date.
Specific tasks:
* Write a unit tests that mimics this situation and verify that the date of the fetch is roughly accurate; fix the bug if there is one
* Correct the row linked above so the date is something sensible (a cqlsh export seems to have problems with this date, so we need to fix it)

<-- end

There is a code change with this patch, it does appear that if robots rejected a link, it would be saved with a FetchTime set to time.Time{}. I changed it to be "Not yet crawled" so that the console can render that message (which is technically true, since if a link was robots excluded, it hasn't been crawled). The mystery is why the date for that first historical reads 1754 -- time.Time{} resolves to the year 1AD, not 1754. I can't figure out a plausible reason that this date is present in the DS. I tried to use the full robots.txt file from the site in question, and wasn't able to get any traction. When I take a look at the links/segments

```
cqlsh:walker> select * from links where dom='banderabulletin.com';

dom | subdom | path | proto | time | err | fnv | getnow | mime | redto_url | robot_ex | stat
---------------------+--------+------+-------+----------------------------------------------------+------+------+--------+------+-----------+----------+------
banderabulletin.com | | / | http | datetime.datetime(1754, 8, 30, 22, 43, 41, 129000) | null | 0 | null | null | null | True | null
banderabulletin.com | | / | http | 1969-12-31 16:00:00-0800 | null | null | null | null | null | null | null

cqlsh:walker> select * from segments where dom='banderabulletin.com';

dom | subdom | path | proto | time
---------------------+--------+------+-------+--------------------------
banderabulletin.com | | / | http | 1969-12-31 16:00:00-0800

```

So we see that there are two fetches of /. What in the world is the first 'time' value (datetime.datetime(1754, 8, 30, 22, 43, 41, 129000)) of the first, why isn't it formatted like a normal date(?), I don't know. Maybe it's just a limitation of the date formatter: i.ee it's a limitation with the date format that can't represent dates before a certain time. But at this point I'm willing to accept that this data got installed in a previous incarnation of walker, and can safely be removed and ignored.

It looks like the domain was claimed 4 days ago. My expectation is when the segment is picked up, the / link should be processed, and then saved with it's robot_ex set to true. Is it reasonable that the link has been sitting in segments for that length of time?

In terms of fixing the data. I feel like we could just remove the 1754 link from the table, and call it a day. Maybe come back periodically for a few weeks to check that 1754 hasn't re-entered the links table for that domain. Comments?