If modified since

Yeah I think your point is correct here. We should be okay if the last fetch was an error or something (in those cases we, when dispatching a link, we should just use the date of the last successful fetch, not the last fetch period), but honoring noindex still has the issue.

I could see us fixing this if we included a field on the links table called `noindex` (and maybe includ `nofollow` while we're at it), so the dispatcher won't use the date from a `noindex` fetch. Do you think that would work? If so I can note it as a possible future todo on our planning page.

For the time being, though, it does seem like a corner case we can ignore.