You can't as this will screw with the walker for sure. 
If you want a dodgy fix for this I would suggest to first call get_category and if that returns none call inactive to reset the timestamps to just below a category but above the periodic cleanup timeout.