Fix incorrect last updated times by not unrefing log_time too early