Instead, I would add up the stream time by summing up ticks on the clock reference device. For every callback, increment the stream time by latency ms. That way even if PortAudio's time information is junk we can still keep track of time. 

Also, using the clock reference device only to count time solves the issue of multiple devices.