Repository: internetarchive/openlibrary

Issue:
Do a full solr reindex with 2021-10 dump
~Blocked by #5493 .~

We're making some change to our solr schema, so want to do a full reindex this month. This will also "deploy" some of the fixes we've made, like #5267

PRs part of this full reindex:
- #5270
- #5568
- #5600
-  ~#3990~ ->> Punting to next one
- [x] Update solr 8.8 -> 8.10

Process
- [x] Run full reindex, fixing any issues
- [x] Start next solr updater to catch-up
- [x] Finish solrupdater catchup
- [x] Connect testing to new solr for testing
- [x] Add haproxy + Connect prod to new solr for a bit of extra testing
- [x] Copy over to solr0
    - [x] Semi-Automate steps in #5493 (Split to separate issue: #5877 )
- [x] ~Commit any docker-compose.production.yml and openlibrary.yml~
    - I just had to change `--no-solr-next` to `--solr-next` for `solr-updater` ; this isn't applicable after the next PR.
- [x] #5878 Remove solr_next from code

## Stakeholders
@jimman2003 


Discussion:
Up on http://testing.openlibrary.org for testing! Would love to get some eyes on it, if folks can confirm the issues below are fixed on testing.openlibrary.org .

Issues addressed:
- #5267
- #4322
- #178
- #714

CC @LeadSongDog @RayBB @ksze @tfmorris @jimman2003 
Stats changes


Type | 10-13 data dump | Prod Solr on ~11-01 | New Solr | (New Solr) - (data dump)| (New Solr) - (Prod Solr)
-- | -- | -- | -- | -- | --
Works | 23736989 | 22224414 | 23736954 | -35 | 1512540
Orphans | 2270861 | 2304061 | 2270854 | -7 | -33207
Authors | 9052252 | 8483628 | 9052192 | -60 | 568564
Subjects | 0 | 2957272 | 3026213 | 3026213 | 68941

Can confirm that 1st issue is fixed
I think 2nd issue looks much better now.
However, some subshelves consistently fail to load for some reason (e.g. 00X):
![image](https://user-images.githubusercontent.com/381556/139811001-8e0957d5-cf8c-4abf-b459-98fbf3cf66c0.png)

But clicking the "View all" link in the subshelves gives me a different page that *does* have books in it:
https://testing.openlibrary.org/search?q=ddc%3A00*+has_fulltext%3Atrue+AND+first_publish_year%3A%5B1985+TO+9998%5D&offset=0&sort=random_2021-11-02T08&limit=20
![image](https://user-images.githubusercontent.com/381556/139811143-d8c218d5-4bcc-48da-8739-576e8d6882c8.png)

Testing some of the examples is somewhat frustrated by #5784, which has arisen in the mean time. 
@ksze Ah, I think the optimization step silently failed, causing search to be slow + sometimes fail (and Library Explorer is likely caching the failed result).

Switching testing back to the other solr while I fix :+1:
Ok, optimized and back on testing.openlibrary.org :+1:
Much better. I believe it has solved #4322.
Awesome! Ok, I've connected production to the new solr, so the changes should no be live on openlibrary.org ! I'm monitoring for perf issues before "cementing" the change and making it permanent.

Resolving patch:
diff --git a/openlibrary/solr/update_work.py b/openlibrary/solr/update_work.py
--- a/openlibrary/solr/update_work.py
+++ b/openlibrary/solr/update_work.py
@@ -603,10 +603,9 @@ def add_list(name, values):
             add_list('publish_year', pub_years)
             add('first_publish_year', min(int(y) for y in pub_years))
 
-        if get_solr_next():
-            number_of_pages_median = pick_number_of_pages_median(editions)
-            if number_of_pages_median:
-                add('number_of_pages_median', number_of_pages_median)
+        number_of_pages_median = pick_number_of_pages_median(editions)
+        if number_of_pages_median:
+            add('number_of_pages_median', number_of_pages_median)
 
         field_map = [
             ('lccn', 'lccn'),
@@ -1699,7 +1698,7 @@ def main(
     profile=False,
     data_provider: Literal['default', 'legacy', 'external'] = "default",
     solr_base: str = None,
-    solr_next=True,
+    solr_next=False,
     update: Literal['update', 'print'] = 'update',
 ):
     """
diff --git a/scripts/new-solr-updater.py b/scripts/new-solr-updater.py
--- a/scripts/new-solr-updater.py
+++ b/scripts/new-solr-updater.py
@@ -251,7 +251,7 @@ def main(
     exclude_edits_containing: str = None,
     ol_url='http://openlibrary.org/',
     solr_url: str = None,
-    solr_next=True,
+    solr_next=False,
     socket_timeout=10,
     load_ia_scans=False,
     commit=True,


Test patch: