Repository: internetarchive/openlibrary

Issue:
Work autocomplete very slow or sometimes 500s
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

### Evidence / Screenshot (if possible)

### Relevant url?
https://openlibrary.org/works/_autocomplete

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to https://openlibrary.org/works/_autocomplete?q=harry&limit=11&timestamp=Thu%20Jul%2027%202023%2013%3A14%3A05%20GMT-0400%20(Eastern%20Daylight%20Saving%20Time)
2. Change the `q` a few times

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: Sometimes 500s
* Expected: Never 500s

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF115
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@seabelis 



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/autocomplete.py b/openlibrary/plugins/worksearch/autocomplete.py
--- a/openlibrary/plugins/worksearch/autocomplete.py
+++ b/openlibrary/plugins/worksearch/autocomplete.py
@@ -37,6 +37,10 @@ def doc_wrap(self, doc: dict):
         if 'name' not in doc:
             doc['name'] = doc.get('title')
 
+    def doc_filter(self, doc: dict) -> bool:
+        """Exclude certain documents"""
+        return True
+
     def GET(self):
         return self.direct_get()
 
@@ -75,10 +79,13 @@ def direct_get(self, fq: list[str] | None = None):
             if fake_doc:
                 docs = [fake_doc]
 
+        result_docs = []
         for d in docs:
-            self.doc_wrap(d)
+            if self.doc_filter(d):
+                self.doc_wrap(d)
+                result_docs.append(d)
 
-        return to_json(docs)
+        return to_json(result_docs)
 
 
 class languages_autocomplete(delegate.page):
@@ -94,15 +101,17 @@ def GET(self):
 
 class works_autocomplete(autocomplete):
     path = "/works/_autocomplete"
-    fq = [
-        'type:work',
-        # Exclude orphaned editions from search results
-        'key:*W',
-    ]
+    fq = ['type:work']
     fl = 'key,title,subtitle,cover_i,first_publish_year,author_name,edition_count'
     olid_suffix = 'W'
     query = 'title:"{q}"^2 OR title:({q}*)'
 
+    def doc_filter(self, doc: dict) -> bool:
+        # Exclude orphaned editions from autocomplete results
+        # Note: Do this here instead of with an `fq=key:*W` for performance
+        # reasons.
+        return doc['key'][-1] == 'W'
+
     def doc_wrap(self, doc: dict):
         doc['full_title'] = doc['title']
         if 'subtitle' in doc:


Test patch:
diff --git a/openlibrary/plugins/worksearch/tests/test_autocomplete.py b/openlibrary/plugins/worksearch/tests/test_autocomplete.py
--- a/openlibrary/plugins/worksearch/tests/test_autocomplete.py
+++ b/openlibrary/plugins/worksearch/tests/test_autocomplete.py
@@ -52,13 +52,18 @@ def test_works_autocomplete():
                     'type': 'work',
                     'title': 'Foo Baz',
                 },
+                {
+                    'key': '/works/OL789M',
+                    'type': 'work',
+                    'title': 'Foo Baz',
+                },
             ]
         }
         result = json.loads(ac.GET().rawtext)
         # assert solr_select called with correct params
         assert mock_solr_select.call_args[0][0] == 'title:"foo"^2 OR title:(foo*)'
         # check kwargs
-        assert mock_solr_select.call_args.kwargs['fq'] == ['type:work', 'key:*W']
+        assert mock_solr_select.call_args.kwargs['fq'] == ['type:work']
         # check result
         assert result == [
             {