- Added robots.txt into repo
- Updated the robots.txt so that it will disallow the crawling of asset folders