Only use Robots' delay if it is shorter than Options' delay

If the robots.txt delay is shorter than the delay specified by the user, we use the robots.txt delay. However, if we request a delay which is longer than that in robots.txt, we will continue to use our longer delay.