Repository: nflverse/nfl_data_py

Issue:
[BUG] NameError when calling import_pbp_data function due to 'HTTPError in exception handling.
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Have you installed the latest development version of the package(s) in question?

- [X] I have installed the latest development version of the package.

### If this is a data issue, have you tried clearing your nflverse cache?

I have cleared my nflverse cache and the issue persists.

### What version of the package do you have?

0.3.3

### Describe the bug

When trying to run the pbp data fetch function (nfl.import_pbp_data(years, columns, downcast=True, cache=False, alt_path=None))
I'm receiving an HTTPError and a NameError is raised as a result

### Reprex

```r
import nfl_data_py as nfl
pbp_data = nfl.import_pbp_data(years=[2024])
```


### Expected Behavior

nfl.import_pbp_data should return a dataframe of the data for the year specified

### nflverse_sitrep

```r
NA
```


### Screenshots

_No response_

### Additional context

Doing some tracing, I believe this may be a data issue with where it's fetching the files from 'nflverse-data'. Could be that this dataset maybe isn't updating correctly and the URL isn't found as result

Discussion:
@cdeez-oss: I am also running into the same problem with 2024 data. However I am on version 0.3.1, I had figured it may be a problem with the older package. @alecglen: Hi there, thanks for the report!

`import_pbp_data()` is erroring currently for 2024 due to the [loss of the pbp_participation dataset](https://x.com/John_B_Edwards/status/1832091502895579354). I'll have a fix in the 0.3.3 release soon, but in the mean time you can avoid the error by calling the function with the optional parameter `include_participation=False`.@Laneville: Thanks @alecglen ! this worked perfectly.@cdeez-oss: Thank you @alecglen! this also works for the older package for anyone else running into the error.@roe-men: also running into this error and tried the above but haven't been able to fix..@alecglen: @roe-men can you paste the full stack trace?@roe-men: An error occurred while pulling the play-by-play data: name 'Error' is not defined
Traceback (most recent call last):
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/nfl_data_py/__init__.py", line 147, in import_pbp_data
    partic = pandas.read_parquet(path)
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/pandas/io/parquet.py", line 667, in read_parquet
    return impl.read(
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/pandas/io/parquet.py", line 396, in read
    handles = get_handle(
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/pandas/io/common.py", line 728, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/pandas/io/common.py", line 384, in _get_filepath_or_buffer
    with urlopen(req_info) as req:
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/pandas/io/common.py", line 289, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/robertomendoza/nfl_folder/play_by_play.py", line 214, in main
    pbp_data = nfl.import_pbp_data(years=years, columns=columns, downcast=True)
  File "/Users/robertomendoza/nfl_folder/venv/lib/python3.9/site-packages/nfl_data_py/__init__.py", line 153, in import_pbp_data
    except Error as e:
NameError: name 'Error' is not defined@alecglen: > pbp_data = nfl.import_pbp_data(years=years, columns=columns, downcast=True)

That 4th line from the bottom is from your code, and I see you're not using the `include_participation=False` argument as described above. Did you perhaps forget to save the file change?@roe-men: that worked. thanks!@alecglen: Fixed in 0.3.3. This will be available via PyPI soon, but can be installed now with the command below if needed.
```
pip install --force-reinstall nfl-data-py@git+https://github.com/nflverse/nfl_data_py.git@v0.3.3
```

Solution patch:
diff --git a/nfl_data_py/__init__.py b/nfl_data_py/__init__.py
index 3c874e1..e66d810 100644
--- a/nfl_data_py/__init__.py
+++ b/nfl_data_py/__init__.py
@@ -10,6 +10,7 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
 import numpy
 import pandas
 import appdirs
+from urllib.error import HTTPError
 
 # module level doc string
 __doc__ = """
@@ -143,13 +144,20 @@ def import_pbp_data(
                 raw = pandas.DataFrame(data)
                 raw['season'] = year
                 
-                if all([include_participation, year >= 2016, not cache]):
+
+                if include_participation and not cache:
                     path = r'https://github.com/nflverse/nflverse-data/releases/download/pbp_participation/pbp_participation_{}.parquet'.format(year)
-                    partic = pandas.read_parquet(path)
-                    raw = raw.merge(partic,
-                                    how='left',
-                                    left_on=['play_id','game_id'],
-                                    right_on=['play_id','nflverse_game_id'])
+
+                    try:
+                        partic = pandas.read_parquet(path)
+                        raw = raw.merge(
+                            partic,
+                            how='left',
+                            left_on=['play_id','game_id'],
+                            right_on=['play_id','nflverse_game_id']
+                        )
+                    except HTTPError:
+                        pass
                 
                 pbp_data.append(raw)
                 print(str(year) + ' done.')
@@ -158,8 +166,10 @@ def import_pbp_data(
                 print(e)
                 print('Data not available for ' + str(year))
     
-    if pbp_data:
-        plays = pandas.concat(pbp_data).reset_index(drop=True)
+    if not pbp_data:
+        return pandas.DataFrame()
+    
+    plays = pandas.concat(pbp_data, ignore_index=True)
     
     # converts float64 to float32, saves ~30% memory
     if downcast:


Test patch:
diff --git a/nfl_data_py/tests/nfl_test.py b/nfl_data_py/tests/nfl_test.py
index b4a0983..779a4c2 100644
--- a/nfl_data_py/tests/nfl_test.py
+++ b/nfl_data_py/tests/nfl_test.py
@@ -9,17 +9,17 @@ import nfl_data_py as nfl
 
 
 class test_pbp(TestCase):
+    pbp = nfl.import_pbp_data([2020])
+
     def test_is_df_with_data(self):
-        s = nfl.import_pbp_data([2020])
-        self.assertIsInstance(s, pd.DataFrame)
-        self.assertTrue(len(s) > 0)
+        self.assertIsInstance(self.pbp, pd.DataFrame)
+        self.assertTrue(len(self.pbp) > 0)
 
     def test_is_df_with_data_thread_requests(self):
         s = nfl.import_pbp_data([2020, 2021], thread_requests=True)
         self.assertIsInstance(s, pd.DataFrame)
         self.assertTrue(len(s) > 0)
 		
-        
     def test_uses_cache_when_cache_is_true(self):
         cache = Path(__file__).parent/f"tmpcache-{random.randint(0, 10000)}"
         self.assertRaises(
@@ -33,6 +33,21 @@ class test_pbp(TestCase):
         self.assertIsInstance(data, pd.DataFrame)
         
         shutil.rmtree(cache)
+
+    def test_includes_participation_by_default(self):
+        self.assertIn("offense_players", self.pbp.columns)
+
+    def test_excludes_participation_when_requested(self):
+        data = nfl.import_pbp_data([2020], include_participation=False)
+        self.assertIsInstance(self.pbp, pd.DataFrame)
+        self.assertTrue(len(self.pbp) > 0)
+        self.assertNotIn("offense_players", data.columns)
+
+    def test_excludes_participation_if_not_available(self):
+        data = nfl.import_pbp_data([2024])
+        self.assertIsInstance(self.pbp, pd.DataFrame)
+        self.assertTrue(len(self.pbp) > 0)
+        self.assertNotIn("offense_players", data.columns)
         
         
 class test_weekly(TestCase):