Fix a timing issue that causes automation to break

@joelrbrandt Can you review this ASAP? I'm afraid this is (part of) Teruko's automation issue that seriously impairs their ability to test.