Only cancel timer if a timeout was set