BUG: bug in to_datetime when passed a np.nan or integer datelike and a format string (GH5863)

closes #5863