Repository: internetarchive/openlibrary

Issue:
Unable to add recently created author to existing record
### Description
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

I was trying to add an recently imported author https://openlibrary.org/authors/OL7631871A/Snorri_Bros.

to an existing work + edition which had no author:
https://openlibrary.org/works/OL16460446W/Barflies

And even though I pasted the author id OL7631871A into the author field, it didn't find the author (I'm guessing because it was not yet indexed in solr), but when I clicked save, it created this no-change edit: https://dev.openlibrary.org/books/OL16491035M/Barflies?b=7&a=6&_compare=Compare&m=diff

and no author was added.
### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Expectation
<!-- What should by happening? What will it look like / how will it behave? -->

I think if the UI can detect an OLID / author ID has been entered, it should save it. The UI looked like it was registering the fact it had an OLID, but nothing happened on save. 

Solr shouldn't be required for a direct id, because it will have the indexing lag (~15mins) and the id should be saved. I'm not too familiar with this code, I think @cdrini you worked on this semi-recently, do you see what the issue is here?

### Details

- **Logged in (Y/N)?**  Y

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cdrini 
@jdlrobson 


Discussion:
The no-change edit on author error has been reported before, relatively recently.

I agree that OL IDs should be detected and handled directly with a regex (optionally a full OL URL too), but that won't help users entering multiple books for a new author (which is how we get lots of duplicate authors). Indexing needs to be [near real-time](https://lucene.apache.org/solr/guide/6_6/near-real-time-searching.html) since it's the basis for so much OL functionality.
#2128 was the issue I was thinking of, but it's not exactly the same -- just another way that an author-less work can be created.
I agree with @tfmorris that the solr indexing needs to be near-real-time.

Is it possible that the search endpoint could hit the database directly in the case of a query matching ol<number> ?

https://openlibrary.org/authors/_autocomplete?q=ol7631871a&limit=11&timestamp=1570831533113

I like the idea, but might not be entirely possible. We'd have to coerce the response from the database to have the same "shape" as the response from solr. Not sure how difficult that would be. (e.g. for authors, `subjects` is a field that exists only on solr; not on the record itself).

I think working towards improving solr might be better then a temporary patch.
Is #2246 going to make Solr near real time?
Modern versions of solr support near real time (NRT) searching ( https://lucene.apache.org/solr/guide/8_3/near-real-time-searching.html ); #2246 updates to a version of solr that supports NRT. I'm not sure if it's setup to do NRT (@tfmorris would know best), but even if it doesn't the update would be a necessary step.
@cdrini, 
It’s great that work is being planned on someday updating solr, but that shouldn’t delay solving this issue. When I create a new author olid and record, it should be Immediately usable in editing work records, even if it takes forever for the name to be findable. We’ve been suffering with this for years with no evident reason. Example:

I updated this work, replacing the author “Hugh Neill“ with a new “Hugh Neill, surgeon”:
![3C95548C-D4E1-421A-A165-8E98CE1C0822](https://user-images.githubusercontent.com/11435431/94043173-85373480-fd9a-11ea-9609-7804171f0032.jpeg)

An hour after creation, the author record is findable by “All” search on olid:
![02916B85-0CCD-4FB4-89F6-FE8785803BF2](https://user-images.githubusercontent.com/11435431/94040974-a185a200-fd97-11ea-82a9-fe33fb21ac34.jpeg)

![F4AC2E43-8E80-449F-88F5-9835D9686264](https://user-images.githubusercontent.com/11435431/94041265-017c4880-fd98-11ea-8e79-ed9551306539.jpeg)

Yet the same record is not found by an “Author” search on olid, nor author name:
![A859C0D5-3633-493C-B400-58572AEF6BD7](https://user-images.githubusercontent.com/11435431/94041060-bfeb9d80-fd97-11ea-9729-5b02a6971008.jpeg)

Worst, even when the olid is known, it can’t be used in editing the work record:
![ACB74E44-ABE7-4397-9F53-140E9D4B979A](https://user-images.githubusercontent.com/11435431/94041107-cb3ec900-fd97-11ea-8fc0-de32c89143a2.jpeg)

Fixing this last should not have had to wait for years on a potential fix to solr. Just accept the offered author_olid.

Resolving patch:
diff --git a/openlibrary/plugins/upstream/addbook.py b/openlibrary/plugins/upstream/addbook.py
--- a/openlibrary/plugins/upstream/addbook.py
+++ b/openlibrary/plugins/upstream/addbook.py
@@ -912,7 +912,8 @@ def GET(self):
         solr = get_solr()
 
         q = solr.escape(i.q).strip()
-        if is_work_olid(q.upper()):
+        query_is_key = is_work_olid(q.upper())
+        if query_is_key:
             # ensure uppercase; key is case sensitive in solr
             solr_q = 'key:"/works/%s"' % q.upper()
         else:
@@ -931,12 +932,20 @@ def GET(self):
         # exclude fake works that actually have an edition key
         docs = [d for d in data['docs'] if d['key'][-1] == 'W']
 
+        if query_is_key and not docs:
+            # Grumble! Work not in solr yet. Create a dummy.
+            key = '/works/%s' % q.upper()
+            work = web.ctx.site.get(key)
+            if work:
+                docs = [work.as_fake_solr_record()]
+
         for d in docs:
             # Required by the frontend
             d['name'] = d['key'].split('/')[-1]
             d['full_title'] = d['title']
             if 'subtitle' in d:
                 d['full_title'] += ": " + d['subtitle']
+
         return to_json(docs)
 
 class authors_autocomplete(delegate.page):
@@ -949,8 +958,8 @@ def GET(self):
         solr = get_solr()
 
         q = solr.escape(i.q).strip()
-        solr_q = ''
-        if is_author_olid(q.upper()):
+        query_is_key = is_author_olid(q.upper())
+        if query_is_key:
             # ensure uppercase; key is case sensitive in solr
             solr_q = 'key:"/authors/%s"' % q.upper()
         else:
@@ -967,6 +976,13 @@ def GET(self):
         data = solr.select(solr_q, **params)
         docs = data['docs']
 
+        if query_is_key and not docs:
+            # Grumble! Must be a new author. Fetch from db, and build a "fake" solr resp
+            key = '/authors/%s' % q.upper()
+            author = web.ctx.site.get(key)
+            if author:
+                docs = [author.as_fake_solr_record()]
+
         for d in docs:
             if 'top_work' in d:
                 d['works'] = [d.pop('top_work')]
diff --git a/openlibrary/plugins/upstream/models.py b/openlibrary/plugins/upstream/models.py
--- a/openlibrary/plugins/upstream/models.py
+++ b/openlibrary/plugins/upstream/models.py
@@ -522,6 +522,21 @@ def get_work_count(self):
         result = works_by_author(self.get_olid(), rows=0)
         return result.num_found
 
+    def as_fake_solr_record(self):
+        record = {
+            'key': self.key,
+            'name': self.name,
+            'top_subjects': [],
+            'work_count': 0,
+            'type': 'author',
+        }
+        if self.death_date:
+            record['death_date'] = self.death_date
+        if self.birth_date:
+            record['birth_date'] = self.birth_date
+        return record
+
+
 re_year = re.compile(r'(\d{4})$')
 
 
@@ -700,6 +715,15 @@ def get_edition_covers(self):
         covers = [e.get_cover() for e in editions]
         return [c for c in covers if c and int(c.id) not in exisiting]
 
+    def as_fake_solr_record(self):
+        record = {
+            'key': self.key,
+            'title': self.get('title'),
+        }
+        if self.subtitle:
+            record['subtitle'] = self.subtitle
+        return record
+
 class Subject(client.Thing):
     def _get_solr_result(self):
         if not self._solr_result:


Test patch: