Repository: pandas-dev/pandas

Issue:
COMPAT: periods needs coercion to np.int64 in _generate_regular_range
https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L1768

a simple `np.int64(periods)` will suffice

(and tests of course)

This needs to guaranteed to be int64 as a result (e.g. if this is in32 it will have an internal overflow which will make it have a valid but weird result)

COMPAT: periods needs coercion to np.int64 in _generate_regular_range
https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L1768

a simple `np.int64(periods)` will suffice

(and tests of course)

This needs to guaranteed to be int64 as a result (e.g. if this is in32 it will have an internal overflow which will make it have a valid but weird result)



Discussion:
Hey @jreback,

I'd like to try tackling this one as a first PR.  Do you mind giving an example line of code that would cause `_generate_regulate_range` to have an overflow?  

I thought maybe it was [this line](https://github.com/pydata/pandas/blob/c6fda694a3f9150d518a83577890e9f798a83b6c/pandas/tseries/index.py#L1784), but I couldn't find an example.  I tried 

```
pd.DatetimeIndex(start='2005', periods=np.int32(1e9), freq='s')
```

but in my case, `stride` was a Python int, and the result of `periods*stride` is a np.int64, so no overflow.

you actually need a bigger number

try np.iinfo(np.int32) + 1

Sorry, I'm probably being a bit dense here, but are you suggesting I try something like:

```
periods =np.iinfo(np.int32).max + 1
periods = np.int32(periods)
pd.DatetimeIndex(start='2005', periods=periods, freq='s')
```

In that case, the overflow happens on the second line.  I can't think of how to produe an overflow _inside_ `_generate_regular_range`...

look at @jorisvandenbossche example in the linked issue https://github.com/pydata/pandas/pull/8907
iirc that example generates the issue (in the index construction)

Cool, I think I know why I wasn't able to reproduce the bug.   @jorisvandenbossche is on 32-bit Linux and I'm on 64-bit Linux, so for me, `periods` gets promoted to `np.int64` before having a chance to overflow, whereas for @jorisvandenbossche, I think it probably stays as `np.int32`.  

Alright, I'll do your proposed fix now.

Hey @jreback,

I'd like to try tackling this one as a first PR.  Do you mind giving an example line of code that would cause `_generate_regulate_range` to have an overflow?  

I thought maybe it was [this line](https://github.com/pydata/pandas/blob/c6fda694a3f9150d518a83577890e9f798a83b6c/pandas/tseries/index.py#L1784), but I couldn't find an example.  I tried 

```
pd.DatetimeIndex(start='2005', periods=np.int32(1e9), freq='s')
```

but in my case, `stride` was a Python int, and the result of `periods*stride` is a np.int64, so no overflow.

you actually need a bigger number

try np.iinfo(np.int32) + 1

Sorry, I'm probably being a bit dense here, but are you suggesting I try something like:

```
periods =np.iinfo(np.int32).max + 1
periods = np.int32(periods)
pd.DatetimeIndex(start='2005', periods=periods, freq='s')
```

In that case, the overflow happens on the second line.  I can't think of how to produe an overflow _inside_ `_generate_regular_range`...

look at @jorisvandenbossche example in the linked issue https://github.com/pydata/pandas/pull/8907
iirc that example generates the issue (in the index construction)

Cool, I think I know why I wasn't able to reproduce the bug.   @jorisvandenbossche is on 32-bit Linux and I'm on 64-bit Linux, so for me, `periods` gets promoted to `np.int64` before having a chance to overflow, whereas for @jorisvandenbossche, I think it probably stays as `np.int32`.  

Alright, I'll do your proposed fix now.


Resolving patch:
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -1781,11 +1781,11 @@ def _generate_regular_range(start, end, periods, offset):
             tz = start.tz
         elif start is not None:
             b = Timestamp(start).value
-            e = b + periods * stride
+            e = b + np.int64(periods) * stride
             tz = start.tz
         elif end is not None:
             e = Timestamp(end).value + stride
-            b = e - periods * stride
+            b = e - np.int64(periods) * stride
             tz = end.tz
         else:
             raise NotImplementedError


Test patch:
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -1909,6 +1909,22 @@ def test_time_loc(self):  # GH8667
             right.iloc[i] *= -10
             tm.assert_series_equal(left, right)
 
+    def test_time_overflow_for_32bit_machines(self):
+        # GH8943.  On some machines NumPy defaults to np.int32 (for example,
+        # 32-bit Linux machines).  In the function _generate_regular_range
+        # found in tseries/index.py, `periods` gets multiplied by `strides`
+        # (which has value 1e9) and since the max value for np.int32 is ~2e9,
+        # and since those machines won't promote np.int32 to np.int64, we get
+        # overflow.
+        periods = np.int_(1000)
+
+        idx1 = pd.date_range(start='2000', periods=periods, freq='S')
+        self.assertEqual(len(idx1), periods)
+
+        idx2 = pd.date_range(end='2000', periods=periods, freq='S')
+        self.assertEqual(len(idx2), periods)
+
+
 class TestPeriodIndex(Base, tm.TestCase):
     _holder = PeriodIndex
     _multiprocess_can_split_ = True