Repository: internetarchive/openlibrary

Issue:
partner_batch_imports.py should not import books published in a future year
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
Per the [Open-Mic topics](https://docs.google.com/document/d/1LEbzsLZ1F9_YIQOoZzO7GoZnG1z-rudhZ9HNtsameTc/edit#heading=h.swvutwwydubf) for the Open Library Community call on 2022-11-29, we should not import partner data for books purporting to be published in a future year, as this is resulting in bad records of books that may never exist.

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
When importing books, `partner_batch_imports.py` does not currently check if the `publish_date` is in a future year when importing. It should. E.g. if an import is attempted in the year 2022, it should not import a book purported to be published in 2023.

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
The proposed solution is to add a check to `batch_import()` in `partner_batch_imports.py` to ensure a book isn't purported to be published in a future year.

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
I will submit a PR to address this.

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles, @cdrini 





Discussion:

Resolving patch:
diff --git a/scripts/partner_batch_imports.py b/scripts/partner_batch_imports.py
--- a/scripts/partner_batch_imports.py
+++ b/scripts/partner_batch_imports.py
@@ -10,10 +10,12 @@
 PYTHONPATH=. python ./scripts/partner_batch_imports.py /olsystem/etc/openlibrary.yml
 """
 
+from collections.abc import Mapping
 import datetime
 import logging
 import os
 import re
+from typing import TypedDict, cast
 
 import requests
 
@@ -244,6 +246,18 @@ def is_low_quality_book(book_item) -> bool:
     )
 
 
+def is_published_in_future_year(book_item: Mapping[str, str | list]) -> bool:
+    """
+    Prevent import of books with a publication after the current year.
+
+    Some import sources have publication dates in a future year, and the likelihood
+    is high that this is bad data. So we don't want to import these.
+    """
+    publish_year = int(cast(str, book_item.get("publish_date", "0")[:4]))  # YYYY
+    this_year = datetime.datetime.now().year
+    return publish_year > this_year
+
+
 def batch_import(path, batch, batch_size=5000):
     logfile = os.path.join(path, 'import.log')
     filenames, offset = load_state(path, logfile)
@@ -262,7 +276,12 @@ def batch_import(path, batch, batch_size=5000):
 
                 try:
                     book_item = csv_to_ol_json_item(line)
-                    if not is_low_quality_book(book_item["data"]):
+                    if not any(
+                        [
+                            is_low_quality_book(book_item["data"]),
+                            is_published_in_future_year(book_item["data"]),
+                        ]
+                    ):
                         book_items.append(book_item)
                 except (AssertionError, IndexError) as e:
                     logger.info(f"Error: {e} from {line}")


Test patch:
diff --git a/scripts/tests/test_partner_batch_imports.py b/scripts/tests/test_partner_batch_imports.py
--- a/scripts/tests/test_partner_batch_imports.py
+++ b/scripts/tests/test_partner_batch_imports.py
@@ -1,5 +1,12 @@
+from datetime import datetime
+
 import pytest
-from ..partner_batch_imports import Biblio, is_low_quality_book
+
+from ..partner_batch_imports import (
+    Biblio,
+    is_low_quality_book,
+    is_published_in_future_year,
+)
 
 csv_row = "USA01961304|0962561851||9780962561856|AC|I|TC||B||Sutra on Upasaka Precepts|The||||||||2006|20060531|Heng-ching, Shih|TR||||||||||||||226|ENG||0.545|22.860|15.240|||||||P|||||||74474||||||27181|USD|30.00||||||||||||||||||||||||||||SUTRAS|BUDDHISM_SACRED BOOKS|||||||||REL007030|REL032000|||||||||HRES|HRG|||||||||RB,BIP,MIR,SYN|1961304|00|9780962561856|67499962||PRN|75422798|||||||BDK America||1||||||||10.1604/9780962561856|91-060120||20060531|||||REL007030||||||"  # noqa: E501
 
@@ -87,3 +94,21 @@ def test_is_low_quality_book():
             'publishers': ['Independently Published'],
         }
     )
+
+
+def test_is_published_in_future_year() -> None:
+    last_year = str(datetime.now().year - 1)
+    last_year_book = {'publish_date': last_year}
+    assert is_published_in_future_year(last_year_book) is False
+
+    this_year = str(datetime.now().year)
+    this_year_book = {'publish_date': this_year}
+    assert is_published_in_future_year(this_year_book) is False
+
+    next_year = str(datetime.now().year + 1)
+    next_year_book = {'publish_date': next_year}
+    assert is_published_in_future_year(next_year_book) is True
+
+    # No publication year
+    no_year_book = {'publish_date': '0'}
+    assert is_published_in_future_year(no_year_book) is False