Fix a bug with delays that caused delayed tasked to get blocked