modified permissions
added crawler tags for different sites