Properly handle datetimes

Part of #4055