add robots and redirect links