Repository: internetarchive/openlibrary

Issue:
Fix BWB Importbot Low Quality records
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
Importbot is importing low-quality records from unspecified source. Titles are public domain reprints (possibly print on demand) with keyword-stuffed titles. 

### Evidence / Screenshot (if possible)
<img width="858" alt="Screenshot 2020-11-24 at 21 31 56" src="https://user-images.githubusercontent.com/17739465/100149057-e87a3b80-2e9d-11eb-9291-b49854aa65b7.png">

Example: https://openlibrary.org/search?q=adventures+of+sherlock+holmes+conan+doyle&mode=everything&page=3

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
Books have valid ISBNs and human editors should be allowed to add manually but these should be blocked from import by importbot.

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@mekarpeles 


Discussion:
Seems like a variation on #4065. Importbot needs to gain some credible tests on the truthiness of source records. I'd look for at least one major library catalog or at least two competing booksellers. The leading digits of the ISBN should map to a real publisher: just search to see if those found at e.g. 9781234* include the publisher asserted in the record.
It’s more blatantly obvious than I thought. Many seem to be showing up in https://openlibrary.org/publishers/Independently_Published apparently with author given as one of many crap publishers 
@mekarpeles @hornc Could you please pause ImportBot until this is addressed? It is still doing it, e.g. OL31436121M a few minutes ago. Just googling 9782072894350 finds several vendors who correctly identify the author as Charles Juliet and the publisher as Gallimard, but instead we get Amazon's craptastic content.
Patrons are reporting these asking for removal.
From a patron:

**This author has 2600 works published this year: https://openlibrary.org/authors/OL7546458A/Gerardus_Blokdyk

However, if you look at their profile on amazon the books are there too with high prices. Some have reviews that are totally fake.
https://www.amazon.com/s?i=stripbooks&rh=p_27%3AGerardus+Blokdyk&s=review-count-rank&qid=1616733962&text=Gerardus+Blokdyk&ref=sr_st_review-count-rank

This is likely just a money laundering or some other scam.
I just don't think these books should be clogging the top search results for things like open source software.https://openlibrary.org/search?title=%22open+source%22&sort=new

Is there any way to deal with this?**
@seabelis 
This came up on Reddit last year too.

Ugh! Cliffnotes for cheating on professional
certification tests, likely outsourced to an ´author’ farm to produce a ´book’ per hour. Dangerous at best, possibly illegal. 

Same outfit in 2008, different spelling: https://openlibrary.org/authors/OL3765576A/Gerard_Blokdijk

Publisher "Art of Service" and variations for many of these. 


@cdrini Would you mind clarifying this? Delaying because it “Needs community discussion” of what, exactly? Is there anyone saying that these records might be good to have, or is it a question of clarifying the criteria for accepting or rejecting new records as useful? 
@mekarpeles wants an explicit list of checks he can use with importbot to exclude books. E.g. a bulleted list of agreed upon exclusions. There's a lot of discussion here, but I think he wants it distilled into a list. Plan is to bring it up as a discussion point with @seabelis today during the community call.
Oh, to close the loop, with the work done here #64 being codified into the spreadsheet here: https://docs.google.com/spreadsheets/d/1mG8Tn-sx73fdcHNAvFCJ-2p_-ULT_5QH90y5LLMpbtY , we'll likely try to convert some of the queries in the spreadsheet to checks that import bot uses to avoid importing matching books so that they don't have to later get bulk deleted!
We might start with excluding any editions with isbn:979* to catch most notebooks, diaries, ruled pads, etc.
I don't think filtering on 979* is a good idea -- 979* is a perfectly valid ISBN range. There are plenty of real books published with those ISBNs, and plenty of notebooks published in the 978* range. 

Chronicle Books LLC publishes childrens books in the 979* range.
KONG & PARK, Incorporated publishes Korean textbooks (in English) in the 979* e.g. https://www.kongnpark.com/books/view/9791186293485

There are many examples of blank notebooks with 978* ISBNs

PR #5924 filters to exclude based on the media type in the source data. There is a whitelist approach too. I'm not sure if that import code is currently running though. https://openlibrary.org/people/bwbimportbot shows no activity since June.

I suggest filtering on the available input metadata to exclude basic non-book types is the best way to improve the quality. 


@hornc 
I stand corrected. I see now that the 979 series has included some book ISBNs since 2008. 
I think @hornc committed some code (which I temporarily commented). @cdrini + @seabelis + @jimman2003 have a list of search queries we can use to filter/facet out known bad patterns of e.g. publisher, title, etc.

Mentioned above: https://docs.google.com/spreadsheets/d/1mG8Tn-sx73fdcHNAvFCJ-2p_-ULT_5QH90y5LLMpbtY
@cdrini and I would be happy to work with any contributors from the community. The main goal is to take the queries from the spreadsheet above and add them as conditional checks at various places here:

https://github.com/internetarchive/openlibrary/blob/master/scripts/partner_batch_imports.py

Perhaps a function called `is_low_quality_book` which checks the book's title, contributors, and other fields to see whether the book passes our quality threshold / criteria
Hi,

My name is Jenni, I am new to open library and internet archive. I would like to work on this issue. Can you assign it to me?
Hi @jennshan, sure let's talk about it! I think the main ideas are listed above...

Can you see if you're able to figure out how to take 1 line of the spreadsheet https://docs.google.com/spreadsheets/d/1mG8Tn-sx73fdcHNAvFCJ-2p_-ULT_5QH90y5LLMpbtY and apply it as a filter against books in https://github.com/internetarchive/openlibrary/blob/master/scripts/partner_batch_imports.py?

Let me know as you have questions!
@mekarpeles Do you have examples of data blocked by this that should have been imported?

https://github.com/internetarchive/openlibrary/blob/8b50277712e6d33da668ce2b4e6991c62341c040/scripts/partner_batch_imports.py#L103-L104

From the commits and comments I can't tell which part of the problem was due to authors and which was due to the item format.
Thanks, I will try to do and let you know.
Hi,

Sorry for the delay, it took some time to update my MAC version then installing the necessary docker software and setting up the code base. I wrote a basic check for a query mentioned in the excel. I will make a pull request tomorrow.

Resolving patch:
diff --git a/scripts/partner_batch_imports.py b/scripts/partner_batch_imports.py
--- a/scripts/partner_batch_imports.py
+++ b/scripts/partner_batch_imports.py
@@ -173,6 +173,10 @@ def csv_to_ol_json_item(line):
     b = Biblio(data)
     return {'ia_id': b.source_id, 'data': b.json()}
 
+def is_low_quality_book(book_item):
+    """check if a book item is of low quality"""
+    return ("notebook" in book_item.title.casefold() and "independently published" in book_item.publisher.casefold())
+
 
 def batch_import(path, batch, batch_size=5000):
     logfile = os.path.join(path, 'import.log')
@@ -191,7 +195,9 @@ def batch_import(path, batch, batch_size=5000):
                     offset = 0
 
                 try:
-                    book_items.append(csv_to_ol_json_item(line))
+                    book_item = csv_to_ol_json_item(line)
+                    if not is_low_quality_book(book_item["data"]):
+                        book_items.append(book_item)
                 except AssertionError as e:
                     logger.info(f"Error: {e} from {line}")
 
@@ -206,7 +212,6 @@ def batch_import(path, batch, batch_size=5000):
                 batch.add_items(book_items)
             update_state(logfile, fname, line_num)
 
-
 def main(ol_config: str, batch_path: str):
     load_config(ol_config)
 


Test patch: