overhaul indexing setting

@Jenso666: Please review whether we should make this changes. In my opinion, specifying if robots are allowed to follow links is unnecessary. That should be done for every link and is already implemented by the message parser.
___
- removed option to set whether search robots are allowed  to follow links on the respective page