When resyncing the time use the -B option to ntpdate

From the man page:

    Force the time to always be slewed using the adjtime() system call,  even
    if  the  measured offset is greater than +-128 ms. The default is to step
    the time using settimeofday() if the offset is  greater  than  +-128  ms.
    Note that, if the offset is much greater than +-128 ms in this case, that
    it can take a long time (hours) to slew the clock to the  correct  value.
    During this time, the host should not be used to synchronize clients.

TL;TR this will cause the kernel to gradually adjust the time over several minutes.  This will avoid issues caused by sudden steps in the time.