Repository: internetarchive/openlibrary

Issue:
Search fails for authors with non-Latin1 characters
or perhaps it's that AKA names aren't being indexed.

For whatever reason, neither of these searches return anything, when they should:

Æschylus https://openlibrary.org/search/authors?q=%C3%86schylus&mode=everything
Αἰσχύλος https://openlibrary.org/search/authors?q=%CE%91%E1%BC%B0%CF%83%CF%87%CF%8D%CE%BB%CE%BF%CF%82&mode=everything

They should both return, at a minimum, https://openlibrary.org/authors/OL74434A/Aeschylus

Unicode folding would take care of (or workaround) the first, but wouldn't help with the second.


Discussion:
A couple more Unicode searches that don't work:

- Kālidāsa https://openlibrary.org/search/authors?q=Ka%CC%84lida%CC%84sa&mode=everything 
   should return https://openlibrary.org/authors/OL528A/Kālidāsa
- Āśāpūrṇā Debī https://openlibrary.org/search/authors?q=A%CC%84s%CC%81a%CC%84pu%CC%84rn%CC%A3a%CC%84+Debi%CC%84&mode=everything 
    should return https://openlibrary.org/authors/OL821A
It's odd that searching for Αἰσχύλος using
https://openlibrary.org/search/authors?q=%CE%91%E1%BC%B0%CF%83%CF%87%CF%8D%CE%BB%CE%BF%CF%82&mode=everything gets no hits while 
https://openlibrary.org/search?q=%CE%91%E1%BC%B0%CF%83%CF%87%CF%8D%CE%BB%CE%BF%CF%82&mode=everything
gets 954 hits. The Greek spelling is listed in the alternative names for https://openlibrary.org/authors/OL74434A/Aeschylus
That's an interesting observation and good clue. It sounds like the alternate names aren't included in the default author search.
The different results from the different search end points is a red herring. 

The non-author search is returning *works* records, not author records. Those works have Αἰσχύλος in the `author_alternative_name` field where it got copied from the author's `alternate_name` field.

I've confirmed that author `alternate_names` doesn't appear to be indexed at all. It's not restricted to non-Latin characters. Neither https://openlibrary.org/search/authors?q=AEschyle nor https://openlibrary.org/search/authors?q=alternate_names%3AEschyle return any results. The general query syntax with field names in the query *does* work, because https://openlibrary.org/search/authors?q=name%3A%22Aeschylus%22+AND+work_count%3A%5B5+TO+*%5D returns the expect results.

I think the alternate_names issue is separate from the non-Latin characters issue because https://openlibrary.org/search/authors?q=Ka%CC%84lida%CC%84sa&mode=everything doesn't return the desired record even though that's the exact name string from the author record cut & paste into the search field. Of course, this could be confounded by the search index update issues, so who knows how many problems are really here.
Whether this is due to #848 or #714, the author dropdown when editing the work also fails to see the alternate names, so when Αἰσχύλος is entered, the dropdown offers to create a new author record rather than suggesting https://openlibrary.org/authors/OL74434A


Hmm, using the search box, an author-search for Αἰσχύλος finds nothing,
https://openlibrary.org/search/authors?q=%CE%91%E1%BC%B0%CF%83%CF%87%CF%8D%CE%BB%CE%BF%CF%82&mode=everything

but an all-search hits the jackpot, finding that string in the "Alternative names" for OL74434A 
https://openlibrary.org/search?q=%CE%91%E1%BC%B0%CF%83%CF%87%CF%8D%CE%BB%CE%BF%CF%82&mode=everything
 
It looks like the ligature in Æschylus is getting converted to the replacement character (?schylus) someplace before the query gets to Solr. Now to track down where...

`solr_1       | INFO: [] webapp=/solr path=/select params={fq=type:author&fq=&q.op=AND&q=?schylus&start=0&rows=100&fl=*&qt=standard&wt=json&sort=work_count+desc} hits=0 status=0 QTime=4 
`
Well, looks like it's not quite so simple.

This works: Suśīlā Desāī https://openlibrary.org/search/authors?q=Su%C5%9B%C4%ABl%C4%81+Des%C4%81%C4%AB
This fails because the default query operator is AND and the second term doesn't match - Basu, Āśīsha. - 
https://openlibrary.org/search/authors?q=Basu%2C+A%CC%84s%CC%81i%CC%84sha.

I'm on the fence as to whether this is worth tracking down given all the other Solr changes pending, so may (re)pursue the Solr schema upgrade instead.

Resolving patch:
diff --git a/openlibrary/plugins/upstream/utils.py b/openlibrary/plugins/upstream/utils.py
--- a/openlibrary/plugins/upstream/utils.py
+++ b/openlibrary/plugins/upstream/utils.py
@@ -1,3 +1,5 @@
+from typing import List, Union, Tuple, Any
+
 import web
 import json
 import babel
@@ -458,12 +460,10 @@ def url_quote(text):
 
 
 @public
-def urlencode(dict_or_list_of_tuples):
+def urlencode(dict_or_list_of_tuples: Union[dict, List[Tuple[str, Any]]]) -> str:
     """
     You probably want to use this, if you're looking to urlencode parameters. This will
     encode things to utf8 that would otherwise cause urlencode to error.
-    :param dict or list dict_or_list_of_tuples:
-    :rtype: basestring
     """
     from six.moves.urllib.parse import urlencode as og_urlencode
     tuples = dict_or_list_of_tuples
diff --git a/openlibrary/plugins/worksearch/code.py b/openlibrary/plugins/worksearch/code.py
--- a/openlibrary/plugins/worksearch/code.py
+++ b/openlibrary/plugins/worksearch/code.py
@@ -5,11 +5,13 @@
 import random
 import re
 import string
+from typing import List, Tuple, Any, Union, Optional
 from unicodedata import normalize
 from json import JSONDecodeError
 import requests
 import web
 from lxml.etree import XML, XMLSyntaxError
+from requests import Response
 from six.moves import urllib
 
 from infogami import config
@@ -370,13 +372,14 @@ def build_q_list(param):
             q_list.append('isbn:(%s)' % (normalize_isbn(param['isbn']) or param['isbn']))
     return (q_list, use_dismax)
 
-def execute_solr_query(url):
-    """
-    Returns a requests.Response or None
-    """
-    stats.begin("solr", url=url)
+
+def execute_solr_query(
+        solr_path: str,
+        params: Union[dict, List[Tuple[str, Any]]]
+) -> Optional[Response]:
+    stats.begin("solr", url=f'{solr_path}?{urlencode(params)}')
     try:
-        response = requests.get(url, timeout=10)
+        response = requests.get(solr_path, params=params, timeout=10)
         response.raise_for_status()
     except requests.HTTPError:
         logger.exception("Failed solr query")
@@ -385,11 +388,15 @@ def execute_solr_query(url):
         stats.end()
     return response
 
-def parse_json_from_solr_query(url):
+
+def parse_json_from_solr_query(
+        solr_path: str,
+        params: Union[dict, List[Tuple[str, Any]]]
+) -> Optional[dict]:
     """
     Returns a json.loaded Python object or None
     """
-    response = execute_solr_query(url)
+    response = execute_solr_query(solr_path, params)
     if not response:
         logger.error("Error parsing empty search engine response")
         return None
@@ -476,9 +483,9 @@ def run_solr_query(param=None, rows=100, page=1, sort=None, spellcheck_count=Non
 
     if 'wt' in param:
         params.append(('wt', param.get('wt')))
-    url = solr_select_url + '?' + urlencode(params)
+    url = f'{solr_select_url}?{urlencode(params)}'
 
-    response = execute_solr_query(url)
+    response = execute_solr_query(solr_select_url, params)
     solr_result = response.content if response else None  # bytes or None
     return (solr_result, url, q_list)
 
@@ -746,8 +753,7 @@ def works_by_author(akey, sort='editions', page=1, rows=100, has_fulltext=False,
     for f in facet_fields:
         params.append(("facet.field", f))
 
-    solr_select = solr_select_url + "?" + urllib.parse.urlencode(params)
-    reply = parse_json_from_solr_query(solr_select)
+    reply = parse_json_from_solr_query(solr_select_url, params)
     if reply is None:
         return web.storage(
             num_found = 0,
@@ -777,8 +783,14 @@ def sorted_work_editions(wkey, json_data=None):
     if json_data:
         reply = json.loads(json_data)
     else:
-        solr_select = solr_select_url + "?version=2.2&q.op=AND&q=%s&rows=10&fl=edition_key&qt=standard&wt=json" % q
-        reply = parse_json_from_solr_query(solr_select)
+        reply = parse_json_from_solr_query(solr_select_url, {
+            'q.op': 'AND',
+            'q': q,
+            'rows': 10,
+            'fl': 'edition_key',
+            'qt': 'standard',
+            'wt': 'json',
+        })
     if reply is None or reply.get('response', {}).get('numFound', 0) == 0:
         return []
     # TODO: Deep JSON structure defense - for now, let it blow up so easier to detect
@@ -787,8 +799,14 @@ def sorted_work_editions(wkey, json_data=None):
 
 def top_books_from_author(akey, rows=5, offset=0):
     q = 'author_key:(' + akey + ')'
-    solr_select = solr_select_url + "?q=%s&start=%d&rows=%d&fl=key,title,edition_count,first_publish_year&wt=json&sort=edition_count+desc" % (q, offset, rows)
-    json_result = parse_json_from_solr_query(solr_select)
+    json_result = parse_json_from_solr_query(solr_select_url, {
+        'q': q,
+        'start': offset,
+        'rows': rows,
+        'fl': 'key,title,edition_count,first_publish_year',
+        'sort': 'edition_count desc',
+        'wt': 'json',
+    })
     if json_result is None:
         return {'books': [], 'total': 0}
     # TODO: Deep JSON structure defense - for now, let it blow up so easier to detect
@@ -817,8 +835,9 @@ def escape_colon(q, vf):
         result += ':' + parts.pop(0)
     return result
 
-def run_solr_search(solr_select):
-    response = execute_solr_query(solr_select)
+
+def run_solr_search(solr_select: str, params: dict):
+    response = execute_solr_query(solr_select, params)
     json_data = response.content if response else None  # bytes or None
     return parse_search_response(json_data)
 
@@ -934,10 +953,23 @@ def GET(self):
     def get_results(self, q, offset=0, limit=100):
         valid_fields = ['key', 'name', 'alternate_names', 'birth_date', 'death_date', 'date', 'work_count']
         q = escape_colon(escape_bracket(q), valid_fields)
+        q_has_fields = ':' in q.replace(r'\:', '')
 
-        solr_select = solr_select_url + "?fq=type:author&q.op=AND&q=%s&fq=&start=%d&rows=%d&fl=*&qt=standard&wt=json" % (web.urlquote(q), offset, limit)
-        solr_select += '&sort=work_count+desc'
-        d = run_solr_search(solr_select)
+        d = run_solr_search(solr_select_url, {
+            'fq': 'type:author',
+            'q.op': 'AND',
+            'q': q,
+            'start': offset,
+            'rows': limit,
+            'fl': '*',
+            'qt': 'standard',
+            'sort': 'work_count desc',
+            'wt': 'json',
+            **({} if q_has_fields else {
+                'defType': 'dismax',
+                'qf': 'name alternate_names'
+            })
+        })
 
         docs = d.get('response', {}).get('docs', [])
         for doc in docs:


Test patch: