Only juggle timezone when with_time is set