Don't lose timeouts

Backwards port of the fix to #2133 

/cc @johnsimons @SimonCropp @andreasohlund