This code is basically what the current teuthology lock does. We should probably fix this in the teuthology lock task as well.