Cleanup after removing user from rolling window