Repository: pvlib/pvlib-python

Issue:
Solar position values are incorrect when using DatetimeIndex with unit='us'
**Describe the bug**
Solar elevation values are wrong when using the default `nrel_numpy` method

**To Reproduce**
Steps to reproduce the behavior:
```
import pvlib

# Example usage:
latitude = 37.7749  # Replace with the desired latitude
longitude = -122.4194  # Replace with the desired longitude

# Create a sample DatetimeIndex (replace with your own datetime index)
date_index = pd.date_range(start='2023-01-01', end='2023-01-05', freq='H')

correct_results = pvlib.solarposition.get_solarposition(
    time=date_index,
    latitude=37.804363,
    longitude=-122.271111,
    method="pyephem"
)

incorrect_results = pvlib.solarposition.get_solarposition(
    time=date_index,
    latitude=37.804363,
    longitude=-122.271111,
) 
```

**Expected behavior**
solar altitude should be on a negative/positive cycle throughout a day

**Screenshots**

**Versions:**
 - ``pvlib.__version__``: 0.10.2
 - ``pandas.__version__``: 2.1.4
 - python: 3.10.13

**Additional context**
Add any other context about the problem here.


There's obviously a workaround, but this was a surprising error, is this known?


Discussion:
I get the same results in both cases.

If the elevation isn't lined up with time, then the datetimes need to be localized with the correct timezone.

Interesting, FWIW this is what the code looks like which generated the error on my end, and you can see the elevation data is off (no negatives throughout a day). Not sure it's worth figuring out reconstructing this exact index, but in case you see why this index might be different:

<img width="953" alt="Screenshot 2023-12-20 at 1 00 52 PM" src="https://github.com/pvlib/pvlib-python/assets/11319429/47833b06-c6fa-414b-8fdf-1e5c4794fb4f">


I think the problem is that the time index in that screenshot has dtype `datetime64[us, US/Pacific]`, but `spa_python` is assuming the input is represented in nanoseconds:

https://github.com/pvlib/pvlib-python/blob/8057393e6eaa57ca1ea60bc79205b94d00b33e42/pvlib/solarposition.py#L369

Here is code to recreate the issue from scratch:

```python
import pandas as pd
import pvlib

for unit in ['ns', 'us']:
    index = pd.date_range('2023-12-09', freq='H', periods=6, tz='US/Pacific', unit=unit)
    sp = pvlib.solarposition.get_solarposition(time=index, latitude=37.804363, longitude=-122.271111)
    print(f'Results for {unit=}:')
    print(sp['elevation'])
```

```
Results for unit='ns':
2023-12-09 00:00:00-08:00   -74.994002
2023-12-09 01:00:00-08:00   -70.408287
2023-12-09 02:00:00-08:00   -60.497812
2023-12-09 03:00:00-08:00   -49.067690
2023-12-09 04:00:00-08:00   -37.255747
2023-12-09 05:00:00-08:00   -25.482336
Freq: H, Name: elevation, dtype: float64
Results for unit='us':
2023-12-09 00:00:00-08:00    13.682861
2023-12-09 01:00:00-08:00    13.692004
2023-12-09 02:00:00-08:00    13.701146
2023-12-09 03:00:00-08:00    13.710287
2023-12-09 04:00:00-08:00    13.719425
2023-12-09 05:00:00-08:00    13.728563
Freq: H, Name: elevation, dtype: float64
```

@patricksheehan, just to be sure, when you run the original reproducer code in https://github.com/pvlib/pvlib-python/issues/1932#issue-2049609747, do you observe the issue?  Or is it only with that `us` index in the notebook?
@kandersolar `unit` was introduced at pandas 2.0.0, I checked this with 1.5.3, maybe that's why I didn't reproduce the error.
Well, I think `ns` is still the default unit in pandas 2.x.  The only way I know how to get an index with `us` units is to explicitly request it (either specifying `unit='us'` when creating the index or converting an existing index with `.astype('datetime64[us]')`).  I'm curious how the index in @patricksheehan's example came to have `us` units :)

`1e9` and `10**9` show up in several places in `solarposition.py`, so we might need to fix this problem in a few different places.
@kandersolar thanks for looking at that, you're totally right that the `us` units is the culprit, the OG code I posted works for me as well (apologies for the bad test code, I always settle at 50% effort for reporting weird things vs not reporting at all).

I'm also honestly not sure how the `us` unit got in there, this is from a client's pickled `DataFrame` so who knows ;).
It's a good heads up for an issue that needs fixed, I don't think we want to limit pvlib to pandas < 2.0.0
Another instance of this bug in the wild (pandas 2.0.2): https://groups.google.com/g/pvlib-python/c/xDcgxTwmGTA
I'd like to do my bit with this issue since I've also experienced this looking to optimize memory usage using rougher precision with `datetime[s]`.

Indeed, [on the Internet](https://stackoverflow.com/questions/15203623/convert-pandas-datetimeindex-to-unix-time) the `// 10**9` solution is clearly favoured, and it really makes sense, since current `pandas.DatetimeIndex` docs state the following as kandersolar noted before.

```
dtype : numpy.dtype or DatetimeTZDtype or str, default None
    Note that the only NumPy dtype allowed is datetime64[ns].
```

1. This would be a reason to consider this a not-PVLIB issue. I'm not really in favour of that.
1. A possible solution that **I haven't tried yet** could be:
    1. Converting to `datetime.datetime` with [pd.DatetimeIndex.to_pydatetime](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DatetimeIndex.to_pydatetime.html#pandas.DatetimeIndex.to_pydatetime). It is not affected by the underlying `dtype`.
    1. Follow [this StackOverflow post](https://stackoverflow.com/questions/19801727/convert-datetime-to-unix-timestamp-and-convert-it-back-in-python). Or [this one](https://stackoverflow.com/questions/8777753/converting-datetime-date-to-utc-timestamp-in-python), mentioned in the former.

1. Another possibility is to warn about the underlying type of `pd.DatetimeIndex` in the [.get_solarposition](https://pvlib-python.readthedocs.io/en/stable/reference/generated/pvlib.solarposition.get_solarposition.html) docs.

1. Regarding doing the conversion manually in PVLIB, it is feasible. I've checked Pandas is capable of doing the right introspection of the `dtype` for non-`ns` data types:

```python
>>> import pandas as pd
>>> from importlib.metadata import version
>>> pd.DatetimeIndex(['2023-09-06']).dtype
dtype('<M8[ns]')
>>> pd.DatetimeIndex(['2023-09-06'], dtype='datetime64[ns]').dtype
dtype('<M8[ns]')
>>> pd.DatetimeIndex(['2023-09-06'], dtype='datetime64[s]').dtype 
dtype('<M8[s]')
>>> pd.DatetimeIndex(pd.date_range('2023-09-03', '2023-09-06', unit='s')).dtype
dtype('<M8[s]')
>>> pd.DatetimeIndex(pd.date_range('2023-09-03', '2023-09-06', unit='s')).to_pydatetime()
array([datetime.datetime(2023, 9, 3, 0, 0),
       datetime.datetime(2023, 9, 4, 0, 0),
       datetime.datetime(2023, 9, 5, 0, 0),
       datetime.datetime(2023, 9, 6, 0, 0)], dtype=object)
>>> version("pandas")
'2.0.2'
```

I advocate for doing no. 2. `.to_pydatetime` exists since Pandas 1.0.0

I may work on this on the future, but if someone else wants, I encourage doing so.
Thanks @echedey-ls!  I think we really need to have pvlib work with `us` and other units.  

The pandas [docs](https://pandas.pydata.org/docs/user_guide/timeseries.html#from-timestamps-to-epoch) suggest this method of calculating unix time, which seems to be working well in my tests:

```python
(stamps - pd.Timestamp("1970-01-01")) // pd.Timedelta("1s")
```

I have a PR in the works with this approach.  The main thing is coming up with comprehensive tests...
Oh, wow, I didn't find that earlier! That's incredibly Pythonic.

Good luck with the tests, that's the worst part.



Patch:
diff --git a/docs/sphinx/source/whatsnew/v0.10.4.rst b/docs/sphinx/source/whatsnew/v0.10.4.rst
index 5c529f6383..997ca4c249 100644
--- a/docs/sphinx/source/whatsnew/v0.10.4.rst
+++ b/docs/sphinx/source/whatsnew/v0.10.4.rst
@@ -12,6 +12,17 @@ Enhancements
 
 Bug fixes
 ~~~~~~~~~
+* Fixed an error in solar position calculations when using
+  :py:class:`pandas.DatetimeIndex` with ``unit`` other than ``'ns'`` (:issue:`1932`).
+  The following functions were affected:
+
+  - :py:class:`~pvlib.modelchain.ModelChain` and :py:func:`~pvlib.solarposition.get_solarposition` with the ``nrel_numpy`` and ``nrel_numba`` methods
+  - :py:func:`~pvlib.solarposition.spa_python`
+  - :py:func:`~pvlib.solarposition.sun_rise_set_transit_spa`
+  - :py:func:`~pvlib.solarposition.nrel_earthsun_distance`
+  - :py:func:`~pvlib.solarposition.hour_angle`
+  - :py:func:`~pvlib.solarposition.sun_rise_set_transit_geometric`
+
 * :py:class:`~pvlib.modelchain.ModelChain` now raises a more useful error when
   ``temperature_model_parameters`` are specified on the passed ``system`` instead of on its ``arrays``. (:issue:`1759`).
 * :py:func:`pvlib.irradiance.ghi_from_poa_driesse_2023` now correctly makes use
@@ -32,6 +43,10 @@ Requirements
 
 Contributors
 ~~~~~~~~~~~~
+* Patrick Sheehan (:ghuser:`patricksheehan`)
+* Echedey Luis (:ghuser:`echedey-ls`)
+* Kevin Anderson (:ghuser:`kandersolar`)
 * Cliff Hansen (:ghuser:`cwhanse`)
 * :ghuser:`matsuobasho`
 * Adam R. Jensen (:ghuser:`AdamRJensen`)
+* Peter Dudfield (:ghuser:`peterdudfield`)
diff --git a/pvlib/solarposition.py b/pvlib/solarposition.py
index 08023daf91..af59184727 100644
--- a/pvlib/solarposition.py
+++ b/pvlib/solarposition.py
@@ -28,9 +28,6 @@
 from pvlib.tools import datetime_to_djd, djd_to_datetime
 
 
-NS_PER_HR = 1.e9 * 3600.  # nanoseconds per hour
-
-
 def get_solarposition(time, latitude, longitude,
                       altitude=None, pressure=None,
                       method='nrel_numpy',
@@ -273,6 +270,19 @@ def _spa_python_import(how):
     return spa
 
 
+def _datetime_to_unixtime(dtindex):
+    # convert a pandas datetime index to unixtime, making sure to handle
+    # different pandas units (ns, us, etc) and time zones correctly
+    if dtindex.tz is not None:
+        # epoch is 1970-01-01 00:00 UTC, but we need to match the input tz
+        # for compatibility with older pandas versions (e.g. v1.3.5)
+        epoch = pd.Timestamp("1970-01-01", tz="UTC").tz_convert(dtindex.tz)
+    else:
+        epoch = pd.Timestamp("1970-01-01")
+
+    return np.array((dtindex - epoch) / pd.Timedelta("1s"))
+
+
 def spa_python(time, latitude, longitude,
                altitude=0, pressure=101325, temperature=12, delta_t=67.0,
                atmos_refract=None, how='numpy', numthreads=4):
@@ -365,7 +375,7 @@ def spa_python(time, latitude, longitude,
         except (TypeError, ValueError):
             time = pd.DatetimeIndex([time, ])
 
-    unixtime = np.array(time.view(np.int64)/10**9)
+    unixtime = _datetime_to_unixtime(time)
 
     spa = _spa_python_import(how)
 
@@ -444,7 +454,7 @@ def sun_rise_set_transit_spa(times, latitude, longitude, how='numpy',
 
     # must convert to midnight UTC on day of interest
     utcday = pd.DatetimeIndex(times.date).tz_localize('UTC')
-    unixtime = np.array(utcday.view(np.int64)/10**9)
+    unixtime = _datetime_to_unixtime(utcday)
 
     spa = _spa_python_import(how)
 
@@ -1000,7 +1010,7 @@ def nrel_earthsun_distance(time, how='numpy', delta_t=67.0, numthreads=4):
         except (TypeError, ValueError):
             time = pd.DatetimeIndex([time, ])
 
-    unixtime = np.array(time.view(np.int64)/10**9)
+    unixtime = _datetime_to_unixtime(time)
 
     spa = _spa_python_import(how)
 
@@ -1377,11 +1387,13 @@ def hour_angle(times, longitude, equation_of_time):
     equation_of_time_spencer71
     equation_of_time_pvcdrom
     """
-    naive_times = times.tz_localize(None)  # naive but still localized
     # hours - timezone = (times - normalized_times) - (naive_times - times)
-    hrs_minus_tzs = 1 / NS_PER_HR * (
-        2 * times.view(np.int64) - times.normalize().view(np.int64) -
-        naive_times.view(np.int64))
+    if times.tz is None:
+        times = times.tz_localize('utc')
+    tzs = np.array([ts.utcoffset().total_seconds() for ts in times]) / 3600
+
+    hrs_minus_tzs = (times - times.normalize()) / pd.Timedelta('1h') - tzs
+
     # ensure array return instead of a version-dependent pandas <T>Index
     return np.asarray(
         15. * (hrs_minus_tzs - 12.) + longitude + equation_of_time / 4.)
@@ -1389,9 +1401,9 @@ def hour_angle(times, longitude, equation_of_time):
 
 def _hour_angle_to_hours(times, hourangle, longitude, equation_of_time):
     """converts hour angles in degrees to hours as a numpy array"""
-    naive_times = times.tz_localize(None)  # naive but still localized
-    tzs = 1 / NS_PER_HR * (
-        naive_times.view(np.int64) - times.view(np.int64))
+    if times.tz is None:
+        times = times.tz_localize('utc')
+    tzs = np.array([ts.utcoffset().total_seconds() for ts in times]) / 3600
     hours = (hourangle - longitude - equation_of_time / 4.) / 15. + 12. + tzs
     return np.asarray(hours)
 
@@ -1405,16 +1417,13 @@ def _local_times_from_hours_since_midnight(times, hours):
     # normalize local, naive times to previous midnight and add the hours until
     # sunrise, sunset, and transit
     return pd.DatetimeIndex(
-        (naive_times.normalize().view(np.int64) +
-         (hours * NS_PER_HR).astype(np.int64)).astype('datetime64[ns]'),
-        tz=tz_info)
+        naive_times.normalize() + pd.to_timedelta(hours, unit='h'), tz=tz_info)
 
 
 def _times_to_hours_after_local_midnight(times):
     """convert local pandas datetime indices to array of hours as floats"""
     times = times.tz_localize(None)
-    hrs = 1 / NS_PER_HR * (
-        times.view(np.int64) - times.normalize().view(np.int64))
+    hrs = (times - times.normalize()) / pd.Timedelta('1h')
     return np.array(hrs)
 
 


Test patch:
diff --git a/pvlib/tests/conftest.py b/pvlib/tests/conftest.py
index d74dfb35b4..355d7f0a5a 100644
--- a/pvlib/tests/conftest.py
+++ b/pvlib/tests/conftest.py
@@ -174,6 +174,11 @@ def has_numba():
 requires_pysam = pytest.mark.skipif(not has_pysam, reason="requires PySAM")
 
 
+has_pandas_2_0 = Version(pd.__version__) >= Version("2.0.0")
+requires_pandas_2_0 = pytest.mark.skipif(not has_pandas_2_0,
+                                         reason="requires pandas>=2.0.0")
+
+
 @pytest.fixture()
 def golden():
     return Location(39.742476, -105.1786, 'America/Denver', 1830.14)
diff --git a/pvlib/tests/test_solarposition.py b/pvlib/tests/test_solarposition.py
index 17870de27e..63e32a4c3c 100644
--- a/pvlib/tests/test_solarposition.py
+++ b/pvlib/tests/test_solarposition.py
@@ -12,8 +12,9 @@
 from pvlib.location import Location
 from pvlib import solarposition, spa
 
-from .conftest import requires_ephem, requires_spa_c, requires_numba
-
+from .conftest import (
+    requires_ephem, requires_spa_c, requires_numba, requires_pandas_2_0
+)
 
 # setup times and locations to be tested.
 times = pd.date_range(start=datetime.datetime(2014, 6, 24),
@@ -717,6 +718,119 @@ def test_sun_rise_set_transit_geometric(expected_rise_set_spa, golden_mst):
                        atol=np.abs(expected_transit_error).max())
 
 
+@pytest.mark.parametrize('tz', [None, 'utc', 'US/Eastern'])
+def test__datetime_to_unixtime(tz):
+    # for pandas < 2.0 where "unit" doesn't exist in pd.date_range. note that
+    # unit of ns is the only option in pandas<2, and the default in pandas 2.x
+    times = pd.date_range(start='2019-01-01', freq='h', periods=3, tz=tz)
+    expected = times.view(np.int64)/10**9
+    actual = solarposition._datetime_to_unixtime(times)
+    np.testing.assert_equal(expected, actual)
+
+
+@requires_pandas_2_0
+@pytest.mark.parametrize('unit', ['ns', 'us', 's'])
+@pytest.mark.parametrize('tz', [None, 'utc', 'US/Eastern'])
+def test__datetime_to_unixtime_units(unit, tz):
+    kwargs = dict(start='2019-01-01', freq='h', periods=3)
+    times = pd.date_range(**kwargs, unit='ns', tz='UTC')
+    expected = times.view(np.int64)/10**9
+
+    times = pd.date_range(**kwargs, unit=unit, tz='UTC').tz_convert(tz)
+    actual = solarposition._datetime_to_unixtime(times)
+    np.testing.assert_equal(expected, actual)
+
+
+@requires_pandas_2_0
+@pytest.mark.parametrize('method', [
+    'nrel_numpy',
+    'ephemeris',
+    pytest.param('pyephem', marks=requires_ephem),
+    pytest.param('nrel_numba', marks=requires_numba),
+    pytest.param('nrel_c', marks=requires_spa_c),
+])
+@pytest.mark.parametrize('tz', [None, 'utc', 'US/Eastern'])
+def test_get_solarposition_microsecond_index(method, tz):
+    # https://github.com/pvlib/pvlib-python/issues/1932
+
+    kwargs = dict(start='2019-01-01', freq='H', periods=24, tz=tz)
+
+    index_ns = pd.date_range(unit='ns', **kwargs)
+    index_us = pd.date_range(unit='us', **kwargs)
+
+    sp_ns = solarposition.get_solarposition(index_ns, 40, -80, method=method)
+    sp_us = solarposition.get_solarposition(index_us, 40, -80, method=method)
+
+    assert_frame_equal(sp_ns, sp_us, check_index_type=False)
+
+
+@requires_pandas_2_0
+@pytest.mark.parametrize('tz', [None, 'utc', 'US/Eastern'])
+def test_nrel_earthsun_distance_microsecond_index(tz):
+    # https://github.com/pvlib/pvlib-python/issues/1932
+
+    kwargs = dict(start='2019-01-01', freq='H', periods=24, tz=tz)
+
+    index_ns = pd.date_range(unit='ns', **kwargs)
+    index_us = pd.date_range(unit='us', **kwargs)
+
+    esd_ns = solarposition.nrel_earthsun_distance(index_ns)
+    esd_us = solarposition.nrel_earthsun_distance(index_us)
+
+    assert_series_equal(esd_ns, esd_us, check_index_type=False)
+
+
+@requires_pandas_2_0
+@pytest.mark.parametrize('tz', [None, 'utc', 'US/Eastern'])
+def test_hour_angle_microsecond_index(tz):
+    # https://github.com/pvlib/pvlib-python/issues/1932
+
+    kwargs = dict(start='2019-01-01', freq='H', periods=24, tz=tz)
+
+    index_ns = pd.date_range(unit='ns', **kwargs)
+    index_us = pd.date_range(unit='us', **kwargs)
+
+    ha_ns = solarposition.hour_angle(index_ns, -80, 0)
+    ha_us = solarposition.hour_angle(index_us, -80, 0)
+
+    np.testing.assert_equal(ha_ns, ha_us)
+
+
+@requires_pandas_2_0
+@pytest.mark.parametrize('tz', ['utc', 'US/Eastern'])
+def test_rise_set_transit_spa_microsecond_index(tz):
+    # https://github.com/pvlib/pvlib-python/issues/1932
+
+    kwargs = dict(start='2019-01-01', freq='H', periods=24, tz=tz)
+
+    index_ns = pd.date_range(unit='ns', **kwargs)
+    index_us = pd.date_range(unit='us', **kwargs)
+
+    rst_ns = solarposition.sun_rise_set_transit_spa(index_ns, 40, -80)
+    rst_us = solarposition.sun_rise_set_transit_spa(index_us, 40, -80)
+
+    assert_frame_equal(rst_ns, rst_us, check_index_type=False)
+
+
+@requires_pandas_2_0
+@pytest.mark.parametrize('tz', [None, 'utc', 'US/Eastern'])
+def test_rise_set_transit_geometric_microsecond_index(tz):
+    # https://github.com/pvlib/pvlib-python/issues/1932
+
+    kwargs = dict(start='2019-01-01', freq='H', periods=24, tz=tz)
+
+    index_ns = pd.date_range(unit='ns', **kwargs)
+    index_us = pd.date_range(unit='us', **kwargs)
+
+    args = (40, -80, 0, 0)
+    rst_ns = solarposition.sun_rise_set_transit_geometric(index_ns, *args)
+    rst_us = solarposition.sun_rise_set_transit_geometric(index_us, *args)
+
+    for times_ns, times_us in zip(rst_ns, rst_us):
+        # can't use a fancy assert function here since the units are different
+        assert all(times_ns == times_us)
+
+
 # put numba tests at end of file to minimize reloading
 
 @requires_numba