Repository: pvlib/pvlib-python

Issue:
SolarAnywhere File -- pvlib.iotools.read_tmy3 Bug
@AdamRJensen 

There's a bug report for reading SolarAnywhere Files with using the pvlib.iotools.read_tmy3 function. This bug is in the TMY3 file (I think?)


![TMY3](https://github.com/pvlib/pvlib-python/assets/74630912/1f85b014-a40a-42af-9c07-76e51ccc606e)



Discussion:
Interesting, did this come up during the 2023 PVPMC workshop? These functions are meant to read the legacy TMY files from the [NSRDB archives](https://nsrdb.nrel.gov/data-sets/archives), but because SolarAnywhere TGY files are similar, you may also be able to read them with `pvlib.iotools.read_tmy3()` but you might need to edit the SolarAnywhere file to match the NSRDB TMY3 file format more closely. Can you paste the exact error message you got when trying to read the SolarAnywhere file?
The error appeared when one of the hackathon participants tried to read [this SolarAnywhere](https://raw.githubusercontent.com/PVSC-Python-Tutorials/PVPMC_2023/main/data/SolarAnywhere%20Typical%20DNI%20Year%20Lat_40_75%20Lon_-111_85%20TMY3%20format.csv) file.

There was an unknown character in the `meta['USAF']` value that couldn't be converted to an int.

However, I cannot reproduce the error.  I wonder if it was caused by the way the participant saved that file locally.

I can't reproduce the error either.

@PGRenewables what version of pvlib were you using? You can check that by writing ``pvlib.__version__``.
Also, if you could upload the file you saved locally, that would be great.
Regardless, encoding problems are nightmarish at times, I wonder if we should dedicate a short troubleshooting section to it as perhaps we've already done for time zones?
@AdamRJensen 

PVLib Version  0.9.2

[PVLib Tutorial.zip](https://github.com/pvlib/pvlib-python/files/11456496/PVLib.Tutorial.zip)

I included the files that I was working with in this zip file.
@PGRenewables Thanks!  Can you upload the data file itself as well?  Since they can't reproduce the error with the original file, a next step is to check whether there is some difference between the original data file and the one you're using.  
Hi @kandersolar , this is the file that I was using. Thank you so much! 


[SolarAnywhere Typical DNI Year Lat_40_75 Lon_-111_85_TMY3_format.csv](https://github.com/pvlib/pvlib-python/files/11466930/SolarAnywhere.Typical.DNI.Year.Lat_40_75.Lon_-111_85_TMY3_format.csv)

`print(open(filename, 'r').read(5))` shows that the file @PGRenewables just uploaded differs from the original in that it starts with `ï»¿`, which google says is a [Byte Order Mark](https://en.wikipedia.org/wiki/Byte_order_mark#Byte_order_marks_by_encoding) for UTF-8.  I also notice that the date format inside the CSV data is different in this file (`01/01/2004` versus `1/1/2004`).  I speculate that the data got read into some tool that understands CSV (Excel?) and then re-exported before getting used with pvlib?

The internet suggests using `encoding='utf-8-sig'` for files with this BOM.  We could try to handle this case, but I'm nervous that `read_tmy3` will turn into a rat's nest if it keeps trying to handle encoding issues itself.  I still think exposing `encoding` as an optional parameter is worth considering (https://github.com/pvlib/pvlib-python/pull/1494#issuecomment-1194011363).

Patch:
diff --git a/pvlib/iotools/tmy.py b/pvlib/iotools/tmy.py
--- a/pvlib/iotools/tmy.py
+++ b/pvlib/iotools/tmy.py
@@ -24,7 +24,8 @@
 }
 
 
-def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
+def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None,
+              encoding=None):
     """Read a TMY3 file into a pandas dataframe.
 
     Note that values contained in the metadata dictionary are unchanged
@@ -50,6 +51,11 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
         If ``True``, apply standard names to TMY3 columns. Typically this
         results in stripping the units from the column name.
         Cannot be used in combination with ``map_variables``.
+    encoding : str, optional
+        Encoding of the file. For files that contain non-UTF8 characters it may
+        be necessary to specify an alternative encoding, e.g., for
+        SolarAnywhere TMY3 files the encoding should be 'iso-8859-1'. Users
+        may also consider using the 'utf-8-sig' encoding.
 
     Returns
     -------
@@ -58,7 +64,7 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
     data : DataFrame
         A pandas dataframe with the columns described in the table
         below. For more detailed descriptions of each component, please
-        consult the TMY3 User's Manual ([1]_), especially tables 1-1
+        consult the TMY3 User's Manual [1]_, especially tables 1-1
         through 1-6.
 
     metadata : dict
@@ -187,14 +193,12 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
     """  # noqa: E501
     head = ['USAF', 'Name', 'State', 'TZ', 'latitude', 'longitude', 'altitude']
 
-    try:
-        with open(str(filename), 'r') as fbuf:
-            firstline, data = _parse_tmy3(fbuf)
-    # SolarAnywhere files contain non-UTF8 characters and may require
-    # encoding='iso-8859-1' in order to be parsed
-    except UnicodeDecodeError:
-        with open(str(filename), 'r', encoding='iso-8859-1') as fbuf:
-            firstline, data = _parse_tmy3(fbuf)
+    with open(str(filename), 'r', encoding=encoding) as fbuf:
+        # header information on the 1st line (0 indexing)
+        firstline = fbuf.readline()
+        # use pandas to read the csv file buffer
+        # header is actually the second line, but tell pandas to look for
+        data = pd.read_csv(fbuf, header=0)
 
     meta = dict(zip(head, firstline.rstrip('\n').split(",")))
     # convert metadata strings to numeric types
@@ -206,8 +210,10 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
 
     # get the date column as a pd.Series of numpy datetime64
     data_ymd = pd.to_datetime(data['Date (MM/DD/YYYY)'], format='%m/%d/%Y')
+    # extract minutes
+    minutes = data['Time (HH:MM)'].str.split(':').str[1].astype(int)
     # shift the time column so that midnite is 00:00 instead of 24:00
-    shifted_hour = data['Time (HH:MM)'].str[:2].astype(int) % 24
+    shifted_hour = data['Time (HH:MM)'].str.split(':').str[0].astype(int) % 24
     # shift the dates at midnight (24:00) so they correspond to the next day.
     # If midnight is specified as 00:00 do not shift date.
     data_ymd[data['Time (HH:MM)'].str[:2] == '24'] += datetime.timedelta(days=1)  # noqa: E501
@@ -225,7 +231,8 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
         data_ymd.iloc[-1] = data_ymd.iloc[-1].replace(year=coerce_year+1)
     # NOTE: as of pvlib-0.6.3, min req is pandas-0.18.1, so pd.to_timedelta
     # unit must be in (D,h,m,s,ms,us,ns), but pandas>=0.24 allows unit='hour'
-    data.index = data_ymd + pd.to_timedelta(shifted_hour, unit='h')
+    data.index = data_ymd + pd.to_timedelta(shifted_hour, unit='h') \
+        + pd.to_timedelta(minutes, unit='min')
     # shouldnt' specify both recolumn and map_variables
     if recolumn is not None and map_variables is not None:
         msg = "`map_variables` and `recolumn` cannot both be specified"
@@ -252,15 +259,6 @@ def read_tmy3(filename, coerce_year=None, map_variables=None, recolumn=None):
     return data, meta
 
 
-def _parse_tmy3(fbuf):
-    # header information on the 1st line (0 indexing)
-    firstline = fbuf.readline()
-    # use pandas to read the csv file buffer
-    # header is actually the second line, but tell pandas to look for
-    data = pd.read_csv(fbuf, header=0)
-    return firstline, data
-
-
 def _recolumn(tmy3_dataframe):
     """
     Rename the columns of the TMY3 DataFrame.
@@ -328,7 +326,7 @@ def read_tmy2(filename):
     data : DataFrame
         A dataframe with the columns described in the table below. For a
         more detailed descriptions of each component, please consult the
-        TMY2 User's Manual ([1]_), especially tables 3-1 through 3-6, and
+        TMY2 User's Manual [1]_, especially tables 3-1 through 3-6, and
         Appendix B.
 
     metadata : dict
@@ -430,6 +428,7 @@ def read_tmy2(filename):
     ----------
     .. [1] Marion, W and Urban, K. "Wilcox, S and Marion, W. "User's Manual
        for TMY2s". NREL 1995.
+       :doi:`10.2172/87130`
     """  # noqa: E501
     # paste in the column info as one long line
     string = '%2d%2d%2d%2d%4d%4d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%4d%1s%1d%2d%1s%1d%2d%1s%1d%4d%1s%1d%4d%1s%1d%3d%1s%1d%4d%1s%1d%3d%1s%1d%3d%1s%1d%4d%1s%1d%5d%1s%1d%10d%3d%1s%1d%3d%1s%1d%3d%1s%1d%2d%1s%1d'  # noqa: E501


Test patch:
diff --git a/pvlib/tests/iotools/test_tmy.py b/pvlib/tests/iotools/test_tmy.py
--- a/pvlib/tests/iotools/test_tmy.py
+++ b/pvlib/tests/iotools/test_tmy.py
@@ -121,7 +121,8 @@ def test_solaranywhere_tmy3(solaranywhere_index):
     # The SolarAnywhere TMY3 format specifies midnight as 00:00 whereas the
     # NREL TMY3 format utilizes 24:00. The SolarAnywhere file is therefore
     # included to test files with  00:00 timestamps are parsed correctly
-    data, meta = tmy.read_tmy3(TMY3_SOLARANYWHERE, map_variables=False)
+    data, meta = tmy.read_tmy3(TMY3_SOLARANYWHERE, encoding='iso-8859-1',
+                               map_variables=False)
     pd.testing.assert_index_equal(data.index, solaranywhere_index)
     assert meta['USAF'] == 0
     assert meta['Name'] == 'Burlington  United States'