Repository: internetarchive/openlibrary

Issue:
 BWB imports have invented publication dates
The number of editions published on January 1st has skyrocketed recently and the apparent cause is either BetterWorldBooks (BWB) metadata or a bug in the BWB importer.

### Evidence / Screenshot (if possible)
![Screen Shot 2023-04-03 at 6 58 15 PM](https://user-images.githubusercontent.com/82178/229651428-3fdd046e-7c6d-49e7-9ca0-ac6279d8acef.png)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce

1. Go to ... almost any recently imported edition
2. Do ... check the recorded publication date against the actual date from a reliable source

* Actual: Publication years are getting assumed to be dates of Jan. 1
* Expected: Publication years are left as just years


### Proposal & Constraints
Stop importing bad data from BetterWorldBooks (BWB)

### Stakeholders
@mekarpeles @hornc 


Discussion:
Hi, @jimchamp! Our team (@yujiezh9 and I) are students from a software engineering course, and both of us have experience in web application development and successfully build and run this project locally. We are wondering whether we can be assigned to this task, thank you!
@AGoodName244 this is a data issue that I don't quite have an approach for right now, but will be addressed as we revamp our import pipeline this year.
Maybe you'd be interested in #7755, which has a solution outlined?  If so, comment on that issue and somebody will assign you.
> @AGoodName244 this is a data issue that I don't quite have an approach for right now, but will be addressed as we revamp our import pipeline this year. Maybe you'd be interested in #7755, which has a solution outlined? If so, comment on that issue and somebody will assign you.

Thank you for your response and suggestion. We will certainly take a look at issue #7755. Before your reply, we conducted some research into the data issue and found some potential clues, although we haven't yet solved it. (Based on development version)
1. Specifically, we examined the metadata extractor API from Amazon and Better World Books under openlibrary/core/vendor.py. We found that the metadata from Amazon includes the publish_date, while the metadata from Better World Books does not.
2. Furthermore, we investigated a related book in this issue, and in issue #7756, the book "Northern fishes" mentioned in this issue does have a publish date of January 1st, 1960 according to the Amazon page. However, "Northern fishes, with special reference to the upper Mississippi valley," was also set with a January 1st publish date, despite being abnormal. We discovered that the source from Amazon for this book does not provide an exact publication date.
3. Upon examining the code in vendors.py, we noticed that the serialize function provides a way to parse the publication_date. Code
![fbdb33e23f2e2e27fa9412e4b424a6f](https://user-images.githubusercontent.com/60837226/231547428-1498d1a1-3856-4188-960e-abe4857bbd2d.png)
However, when the date is not exactly accurate (for example, a date with only a year), the parse function sets the month and date with some default numbers. 
![7b32034f4d8633f4a171d8c9f80689d](https://user-images.githubusercontent.com/60837226/231548984-74211712-c933-4e40-ac17-32f83f57fa03.png)
As a result, we assume that if the program cannot get an exact date from Amazon, the parser will set a default to publish month and date to the book, although we have not yet tested it to prove it. We hope that this investigation is helpful to the task, and we remain eager to contribute to the project. Please let us know if you have any further guidance or feedback.
@AGoodName244, the code in /core/vendors.py is unrelated to our importer code.  If I recall correctly, `vendors.py` fetches price information for book pages, and is used to create new editions, if needed, when people visit an /isbn/{isbn} page.

The publish dates were parsed correctly from the source import data in each of the cases that you outlined:
| Record | Title | Source Data |
| --- | --- | --- |
| [OL45991226M](https://openlibrary.org/books/OL45991226M) | Northern Fishes | [JSON](https://ia601805.us.archive.org/16/items/bwb_daily_pallets_2020-11-13/DailyPallets__2020-11-13.json) |
| [OL45868829M](https://openlibrary.org/books/OL45868829M) | Northern fishes, with special reference to the upper Mississippi valley | [JSON](https://ia601808.us.archive.org/10/items/bwb_daily_pallets_2021-03-03/DailyPallets__2021-03-03.json) |


> @AGoodName244, the code in /core/vendors.py is unrelated to our importer code. If I recall correctly, `vendors.py` fetches price information for book pages, and is used to create new editions, if needed, when people visit an /isbn/{isbn} page.
> 
> The publish dates were parsed correctly from the source import data in each of the cases that you outlined:
> 
> Record	Title	Source Data
> [OL45991226M](https://openlibrary.org/books/OL45991226M)	Northern Fishes	[JSON](https://ia601805.us.archive.org/16/items/bwb_daily_pallets_2020-11-13/DailyPallets__2020-11-13.json)
> [OL45868829M](https://openlibrary.org/books/OL45868829M)	Northern fishes, with special reference to the upper Mississippi valley	[JSON](https://ia601808.us.archive.org/10/items/bwb_daily_pallets_2021-03-03/DailyPallets__2021-03-03.json)

Thank you so much for the clarification on the issue, and apologies for any confusion caused. After examining the JSON files, we have some concerns about the data that we are curious about. It appears that there may be some discrepancies or mismatches in the data. 
For the JSON file of 
| Record | Title | Source Data |
| -------- | ---- | ------------- | 
[OL45868829M](https://openlibrary.org/books/OL45868829M) | Northern fishes, with special reference to the upper Mississippi valley | [JSON](https://ia601808.us.archive.org/10/items/bwb_daily_pallets_2021-03-03/DailyPallets__2021-03-03.json) |

The book "Managerial Epidemiology" (ASIN: 076373165X) showed a Publication date of 20050101, while the website displays it as May 1, 2005. Additionally, I appreciate your suggestion to focus on issue #7755, and we will certainly look into it. We would be happy to continue our contribution. Thank you again for your support.
For these cases, can we:
Add logic to (explicitly only) BWB importer, ignore 01-01 (and just import the year).

If the years are wrong then the recourse we have is for human/librarians to fix it.

If someone wants to help, the relevant code will be in
1. https://github.com/internetarchive/openlibrary/blob/master/scripts/promise_batch_imports.py (definitely) and
2. https://github.com/internetarchive/openlibrary/blob/master/scripts/partner_batch_imports.py (optionally)
@mekarpeles 
Surely that isn’t scalable. We don’t have that many human contributors.  We know those -01-01 dates are nearly always bogus: what publisher works New Year’s Day?  Just fix ImportBot.
I would not object to importing years only for all cases. 01-01 is just an upstream-enforced date we are importing from elsewhere. Even when a book has an exact date specified (French publishers frequently do this) the correct date is not what we import. Even if the date is not 01-01, the imported exact dates never matches the actual dates specified in the books. Mass-market paperbacks frequently do specify a year and a month. Amazon imports of these are frequently a month off (or sometimes a year off, if it happens to be Dec/Jan). I suspect many of these seemingly arbitrary dates have to do with either the date the item was added to Amazon or the date it went on sale. Neither of these are relevant to us as we are not aiming to be a mirror of Amazon.
Discussed with @judec -- likely an upstream problem with dates coming in as 01-01.

Both of these seem like useful places to investigate (BWB monthly imports and promise item imports):

* https://github.com/internetarchive/openlibrary/blob/master/scripts/partner_batch_imports.py#L90
* https://github.com/internetarchive/openlibrary/blob/master/scripts/promise_batch_imports.py#L39

Resolving patch:
diff --git a/scripts/partner_batch_imports.py b/scripts/partner_batch_imports.py
--- a/scripts/partner_batch_imports.py
+++ b/scripts/partner_batch_imports.py
@@ -121,7 +121,7 @@ def __init__(self, data):
         self.isbn_13 = [self.isbn]
         self.title = data[10]
         self.primary_format = data[6]
-        self.publish_date = data[20][:4]  # YYYY, YYYYMMDD
+        self.publish_date = data[20][:4]  # YYYY
         self.publishers = [data[135]]
         self.weight = data[39]
         self.authors = self.contributors(data)
diff --git a/scripts/promise_batch_imports.py b/scripts/promise_batch_imports.py
--- a/scripts/promise_batch_imports.py
+++ b/scripts/promise_batch_imports.py
@@ -24,14 +24,17 @@
 from openlibrary.core.imports import Batch
 from scripts.solr_builder.solr_builder.fn_to_cli import FnToCLI
 
+
 logger = logging.getLogger("openlibrary.importer.promises")
 
 
-def format_date(date: str) -> str:
-    y = date[0:4]
-    m = date[4:6]
-    d = date[6:8]
-    return f"{y}-{m}-{d}"
+def format_date(date: str, only_year: bool) -> str:
+    """
+    Format date as "yyyy-mm-dd" or only "yyyy"
+
+    :param date: Date in "yyyymmdd" format.
+    """
+    return date[:4] if only_year else f"{date[0:4]}-{date[4:6]}-{date[6:8]}"
 
 
 def map_book_to_olbook(book, promise_id):
@@ -56,8 +59,12 @@ def map_book_to_olbook(book, promise_id):
         'authors': [{"name": book['ProductJSON'].get('Author') or '????'}],
         'publishers': [book['ProductJSON'].get('Publisher') or '????'],
         'source_records': [f"promise:{promise_id}:{sku}"],
-        # format_date adds hyphens between YYYY-MM-DD
-        'publish_date': publish_date and format_date(publish_date) or '????',
+        # format_date adds hyphens between YYYY-MM-DD, or use only YYYY if date is suspect.
+        'publish_date': format_date(
+            date=publish_date, only_year=publish_date[-4:] in ('0000', '0101')
+        )
+        if publish_date
+        else '????',
     }
     if not olbook['identifiers']:
         del olbook['identifiers']


Test patch:
diff --git a/scripts/tests/test_partner_batch_imports.py b/scripts/tests/test_partner_batch_imports.py
--- a/scripts/tests/test_partner_batch_imports.py
+++ b/scripts/tests/test_partner_batch_imports.py
@@ -9,6 +9,7 @@
 )
 
 csv_row = "USA01961304|0962561851||9780962561856|AC|I|TC||B||Sutra on Upasaka Precepts|The||||||||2006|20060531|Heng-ching, Shih|TR||||||||||||||226|ENG||0.545|22.860|15.240|||||||P|||||||74474||||||27181|USD|30.00||||||||||||||||||||||||||||SUTRAS|BUDDHISM_SACRED BOOKS|||||||||REL007030|REL032000|||||||||HRES|HRG|||||||||RB,BIP,MIR,SYN|1961304|00|9780962561856|67499962||PRN|75422798|||||||BDK America||1||||||||10.1604/9780962561856|91-060120||20060531|||||REL007030||||||"  # noqa: E501
+csv_row_with_full_date = "USA01961304|0962561851||9780962561856|AC|I|TC||B||Sutra on Upasaka Precepts|The||||||||20060531|20060531|Heng-ching, Shih|TR||||||||||||||226|ENG||0.545|22.860|15.240|||||||P|||||||74474||||||27181|USD|30.00||||||||||||||||||||||||||||SUTRAS|BUDDHISM_SACRED BOOKS|||||||||REL007030|REL032000|||||||||HRES|HRG|||||||||RB,BIP,MIR,SYN|1961304|00|9780962561856|67499962||PRN|75422798|||||||BDK America||1||||||||10.1604/9780962561856|91-060120||20060531|||||REL007030||||||"  # noqa: E501
 
 
 non_books = [
@@ -37,6 +38,23 @@ def test_sample_csv_row(self):
         }
         assert b.json() == data
 
+    def test_sample_csv_row_with_full_date(self):
+        """Only import the year. The date in the input here is in YYYYMMDD format, but should come out YYYY."""
+        b = Biblio(csv_row.strip().split('|'))
+        data = {
+            'title': 'Sutra on Upasaka Precepts',
+            'isbn_13': ['9780962561856'],
+            'publish_date': '2006',
+            'publishers': ['BDK America'],
+            'weight': '0.545',
+            'authors': [{'name': 'Heng-ching, Shih'}],
+            'pagination': '226',
+            'languages': ['eng'],
+            'subjects': ['Sutras', 'Buddhism, sacred books'],
+            'source_records': ['bwb:9780962561856'],
+        }
+        assert b.json() == data
+
     @pytest.mark.parametrize('input_', non_books)
     def test_non_books_rejected(self, input_):
         data = input_.strip().split('|')
diff --git a/scripts/tests/test_promise_batch_imports.py b/scripts/tests/test_promise_batch_imports.py
new file mode 100644
--- /dev/null
+++ b/scripts/tests/test_promise_batch_imports.py
@@ -0,0 +1,15 @@
+import pytest
+
+from ..promise_batch_imports import format_date
+
+
+@pytest.mark.parametrize(
+    "date, only_year, expected",
+    [
+        ("20001020", False, "2000-10-20"),
+        ("20000101", True, "2000"),
+        ("20000000", True, "2000"),
+    ],
+)
+def test_format_date(date, only_year, expected) -> None:
+    assert format_date(date=date, only_year=only_year) == expected