Allow irc logs and sites gallery to be spidered.