Set last activity only if we use idle timeout.