[Task] robots: updated robots.txt to allow indexing again, but excluded some wordpress specific stuff