BUG: tslib.tz_convert and tslib.tz_convert_single may output different result in DST
State: closed

These functions may return different result in case of DST. There seems to be 2 problems:
- `tslib.tz_convert` checks DST and change `deltas` by adjusting `pos` by 1. If input has time-gaps more than 2 DST spans, result will be incorrect.
- `tslib.tz_convert_single` results incorrect if input is just on DST edge.

```
import pandas as pd
import numpy as np
import datetime
import pytz

idx = pd.date_range('2014-03-01', '2015-01-10', freq='H')

f = lambda x: pd.tslib.tz_convert_single(x, pd.tslib.maybe_get_tz('US/Eastern'), 'UTC')
result = pd.tslib.tz_convert(idx.asi8, pd.tslib.maybe_get_tz('US/Eastern'), 'UTC')
result_single = np.vectorize(f)(idx.asi8)
result[result != result_single]
# [1394370000000000000 1394373600000000000 1394377200000000000 ...,
#  1414918800000000000 1414922400000000000 1414926000000000000]
```
#### Note

Additionally, it was modifed to close #7880 also.


Labels:
name: Performance
description: Memory or execution speed performance
name: Timezones
description: Timezone data dtype

Comments:
Perf result. I think it increases a performance of `inferred_freq` for tz-aware data a bit.

```
...
write_store                                  |  11.6420 |   9.7557 |   1.1934 |
reindex_fillna_pad                           |   1.0404 |   0.8606 |   1.2089 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [7a61a96] : BUG: tslib.tzconvert handles DST incorrectly
Base   [30764bd] : Merge pull request #7783 from lexual/docs_categorical_tidyup
```

can you run the perf test here: https://github.com/pydata/pandas/pull/7652 (and the 'test' in #7633,
which couldn't figure out how to put in a vbench, its basically a `str(df)` with a big period index).

just to make sure the perf is still ok

(your fix looks good though)

OK, done.

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
datetimeindex_normalize                      |   5.6077 |   5.7670 |   0.9724 |

Target [3c8e2b3] : BUG: tslib.tzconvert handles DST incorrectly
Base   [8cd3dd6] : Merge pull request #7652 from jreback/dst_transitions
```

And test in #7633 takes few seconds.

```
import pandas as pd

import numpy as np
period = 1.0 / 2048 * 1e9
freq = pd.datetools.Nano(period)
df = pd.DataFrame({'a': np.random.randn(6e6)}, index=pd.date_range('now', periods=6e6, freq=freq, tz='EST'))
print(df.a)
...
# 2014-07-19 03:08:12.685023438-05:00    0.650118
# 2014-07-19 03:08:12.685511719-05:00   -0.683781
# Freq: 488281N, Name: a, Length: 6000000
# [Finished in 1.9s]
```

Should we add a test for the fall DST transition similar to what you did for spring?  I understand there is not really any specific "spring" code per se, but sometimes strange things popup.  Otherwise the changes look fine.

@rockg Yep, added tests. Could you check whether it is valid, otherwise lmk better one?

That's exactly what I was thinking.

OK, modified test to cover other freqs.

Added the fix for #7880.

@jreback I think it is ready, lmk if anything.

looks fine, what was the problem with #7880 ? (e.g. what fixed it)

Currently, it includes 2 fixes (described in release note)
- Original issue is difference between `tz_convert` and `tz_convert_single` described in problem description.
- Additionally, added the fix for #7880 after receiving the issue. (because #7880 can be fixed by small modification on the same part)

thanks!