Add dynamic allow and disallow of robots