Keep track of the last activity time in each channel