BUG: Fix to_datetime to properly deal with tz offsets #3944

Currently for certain formats of datetime strings, the tz offset will
just be ignored.