[TASK] Applying AU patches. Recursive apply robot-settings

Hi Dmitry
I have a patch to make the changes to the robots-flags recursive. This is currently in use on AU.dk (Aarhus University) in Denmark, and has been in use for a while. If you find it useful and acceptable the patch is here.