Refactoring and also added 4xx and 5xx for crawlers