Fix Time#change when used with :usec and offset times

This is a follow-up of #16758 and the branch is based on it.

Fixes #16759.