Repository: internetarchive/openlibrary

Issue:
Internal Server Error attempting to merge authors with Unicode names
Attempting to merge the more than one dozen duplicate records for Martín de Riquer, conde de Casa Dávalos results in an Internal Server Error and the merge doesn't happen.

### Evidence / Screenshot (if possible)
![Screen Shot 2020-12-06 at 7 47 34 PM](https://user-images.githubusercontent.com/82178/101298455-092a8580-37fc-11eb-9b01-9e280a82f608.png)

### Relevant url?
This is the URL in the browser address bar: https://openlibrary.org/authors/merge?key=OL134443A&key=OL4290954A&key=OL4577335A&key=OL345883A&key=OL627912A&key=OL3214779A&key=OL4234672A&key=OL5311122A&key=OL6604921A&key=OL5053035A&key=OL4396640A&key=OL1667264A&key=OL4409441A&key=OL4253695A&key=OL4248254A&key=OL4234671A&key=OL2375196A&key=OL6343225A&key=OL6384132A&key=OL5345717A&key=OL271803A&key=OL7289138A&key=OL5981870A&key=OL6824492A&key=OL4490320A&key=OL3094412A&key=OL4623608A&key=OL4707454A&key=OL4707453A&key=OL489695A&debug=true
but it's not the actual URL that gives the error, which is apparently buried in a POST somewhere, so it's not accessible 

merge=true&master=OL134443A&merge_key=OL134443A&merge_key=OL4290954A&merge_key=OL4577335A&merge_key=OL5311122A&merge_key=OL1667264A&merge_key=OL4409441A&merge_key=OL4234671A&merge_key=OL5981870A&merge_key=OL4490320A&merge_key=OL4623608A&merge_key=OL4707453A

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ... the url above
2. Do ... the selection of all 15 duplicates & click Merge

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: Internal Server Error, discarding all the work in making the selections
* Expected: merge the 15 duplicate records into one

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/upstream/merge_authors.py#L283
This one too. https://openlibrary.org/authors/merge?key=OL48458A&key=OL7875067A&key=OL7465278A&key=OL4966610A&key=OL5545241A&key=OL6345400A&key=OL2816393A&key=OL6399631A&key=OL6462896A&key=OL7872426A&key=OL7924924A&key=OL8248718A&key=OL1674883A&key=OL5777954A&key=OL2054663A&key=OL2098941A
Another https://openlibrary.org/authors/merge?key=OL640098A&key=OL5450170A&key=OL2833147A&key=OL2345255A&key=OL249916A&key=OL8278962A&key=OL3599758A&key=OL6557211A&key=OL6228601A&key=OL7551973A&key=OL1729497A&key=OL3144881A&key=OL8644521A
This smells like a symptom of an old issue. Variant spellings of author names, particularly with similar-looking accentuation, are often mangled.
Compare
{"name": "Moma Dimi\u0107", "personal_name": "Moma Dimic\u0301", "key": "/authors/OL640098A", "type": {"key": "/type/author"}, "alternate_names": ["Moma Dimic"], "latest_revision": 3, "revision": 3, "created": {"type": "/type/datetime", "value": "2008-04-01T03:28:50.625462"}, "last_modified": {"type": "/type/datetime", "value": "2021-01-04T15:59:08.543406"}}
Versus
{"name": "Moma Dimi\u0107", "personal_name": "Moma Dimi\u0107", "key": "/authors/OL5450170A", "type": {"key": "/type/author"}, "latest_revision": 2, "revision": 2, "created": {"type": "/type/datetime", "value": "2008-10-15T20:52:53.562919"}, "last_modified": {"type": "/type/datetime", "value": "2021-01-04T15:54:46.599110"}}

These two author records show the same “name” but different “personal_name”, which for some reason causes the merge to fail with “Internal server error”

Similarly, compare
{"name": "Andre\u0301 Gide", "personal_name": "Andre\u0301 Gide", "death_date": "1951", "alternate_names": ["Andr\u00e9 Gide", "Andr\u00e9 Paul Guillaume Gide", "Andre Gide", "Andre. Gide", "Gide, Andre, Andre gide", "And\u0155e Gide", "Andre? Gide", "Andr\u02b9e Gide", "A. Gide", "Andre\u0301 Paul Guillaume Gide", "A. P. G. Gide"], "created": {"type": "/type/datetime", "value": "2008-04-01T03:28:50.625462"}, "photos": [6582636, 7273611], "last_modified": {"type": "/type/datetime", "value": "2020-09-27T08:03:21.623046"}, "latest_revision": 11, "key": "/authors/OL48458A", "birth_date": "1869", "revision": 11, "type": {"key": "/type/author"}, "remote_ids": {"viaf": "96973963", "wikidata": "Q47484", "isni": "0000000121445648"}}
Versus
{"created": {"type": "/type/datetime", "value": "2020-05-08T05:04:48.137513"}, "latest_revision": 1, "name": "Andr\u00e9 Gide", "key": "/authors/OL7875067A", "personal_name": "Andr\u00e9 Gide", "birth_date": "1869", "death_date": "1951", "type": {"key": "/type/author"}, "last_modified": {"type": "/type/datetime", "value": "2020-05-08T05:04:48.137513"}, "revision": 1}

After manually tweaking the inconsistent spellings of Martín de Riquer conde de Casa Dávalos to eliminate variant accentuation, the records merged as expected.
Related to https://sentry.archive.org/share/issue/3a2c505b08394882a098a80e2a2d6bc7/
@LeadSongDog If you fix all the test cases by hand, the programmers aren't going to have anything to debug the problem with.
@cclauss That URL goes to a login page which doesn't appear to accept OpenLibrary credentials.
@tfmorris Sure. Here’s another example. See 
https://openlibrary.org/authors/merge?key=OL4963941A&key=OL5636553A&key=OL5568484A

Merging the two Césars here: https://openlibrary.org/authors/merge?key=OL286312A&key=OL193196A&key=OL5331799A&key=OL48705A&key=OL272369A&key=OL442395A&key=OL6176715A&key=OL5051614A&key=OL4211124A&key=OL7966930A&key=OL1532037A&key=OL680450A&key=OL5070419A&key=OL4311841A&key=OL350468A&key=OL1216404A&key=OL2489830A&key=OL5233891A&key=OL391454A&key=OL81538A&key=OL972595A&key=OL1083110A&key=OL5051613A&key=OL597740A&key=OL195408A&key=OL1984572A&key=OL301423A&key=OL1844314A&key=OL4952978A&key=OL493888A&key=OL808750A&key=OL5480322A&key=OL5522586A&key=OL5542711A&key=OL6691980A&key=OL625514A&key=OL2312577A&key=OL5039086A&key=OL5051608A&key=OL368290A&key=OL5102840A&key=OL4183550A&key=OL2475731A&key=OL252509A&key=OL1257575A&key=OL1299293A&key=OL257259A&key=OL5290822A&key=OL1409182A&key=OL72827A&key=OL6014399A&key=OL6834672A&key=OL817804A&key=OL1593681A&key=OL5659722A&key=OL7116084A&key=OL863135A&key=OL190794A&key=OL933511A&key=OL7366398A&key=OL2184396A&key=OL6180249A&key=OL1026302A&key=OL1029350A&key=OL2288275A&key=OL2284058A&key=OL4957816A&key=OL750732A&key=OL360661A&key=OL4971355A&key=OL623687A&key=OL4970053A&key=OL4967050A&key=OL5499074A&key=OL1506504A&key=OL1507918A&key=OL1499302A&key=OL674231A&key=OL5482640A&key=OL1527255A&key=OL1545172A&key=OL1547000A&key=OL5510061A&key=OL677104A&key=OL5543517A&key=OL6255800A&key=OL365539A&key=OL4991338A&key=OL7584856A&key=OL1106129A&key=OL5034639A&key=OL6304580A&key=OL6329284A&key=OL5072009A&key=OL1151634A&key=OL5091061A&key=OL5095777A&key=OL1169653A&key=OL5451046A&key=OL1495185A

Again, looking at the json shows different Unicode in the names between
https://openlibrary.org/authors/OL48705A.json and https://openlibrary.org/authors/OL6176715A.json
These also could be merged: OL29655A,OL7558884A,OL7488497A,OL7465736A,OL7967793A,OL7624788A,OL7562656A

I could merge them with a workaround though. 

1. renaming from OL29655A Stanis**ł**aw Lem to Stanis**l**aw Lem
2. merging them
3. renaming from OL29655A Stanis**l**aw Lem to Stanis**ł**aw Lem
Fails merging the 1st & 3rd authors here: https://openlibrary.org/authors/merge?key=OL619897A&key=OL2908580A&key=OL5551708A&key=OL6373539A&key=OL7507879A&key=OL485083A

But these other Günters merged without problem https://openlibrary.org/authors/OL2908580A/G%C3%BCnter_Kahl
Again note the different encodings of ü in his name as seen at
https://openlibrary.org/authors/OL619897A.json
and at 
https://openlibrary.org/authors/OL5551708A.json

Notes on how to replicate/test this issue
```
["/authors/OL619897A", "/authors/OL2908580A", "/authors/OL5551708A", "/authors/OL6373539A", "/authors/OL7507879A", "/authors/OL485083A"]

# Using JavaScript REPL in FireFox Tools / Web Developer / Web Console...

await fetch('/authors/OL619897A/works.json').then(r => r.json()).then(doc => console.log(doc));

await fetch('/authors/OL619897A/works.json').then(r => r.json()).then(doc => doc.entries.forEach(d => console.log(d.key)));

/works/OL24020240W
/works/OL3632519W
/works/OL3632520W
/works/OL3632521W
/works/OL3632522W
/works/OL3632523W

await Promise.all(["/authors/OL619897A", "/authors/OL2908580A", "/authors/OL5551708A", "/authors/OL6373539A", "/authors/OL7507879A", "/authors/OL485083A"]
.map(key => fetch(`${key}/works.json`).then(r => r.json()).then(doc => doc.entries.forEach(d => console.log(d.key)))));

scripts/copydocs.py -r /works/OL24020240W /works/OL3632519W /works/OL3632520W /works/OL3632521W /works/OL3632522W /works/OL3632523W /works/OL18980446W /works/OL13504315W /works/OL16351582W /works/OL19203064W /works/OL12524517W /works/OL12524518W /works/OL19648085W /works/OL3096834W /works/OL23430581W /works/OL19573986W /works/OL19453717W /works/OL19426221W /works/OL15416211W /works/OL8636583W /works/OL16620320W
```
```js
`/authors/merge?` + ["/authors/OL619897A", "/authors/OL2908580A", "/authors/OL5551708A", "/authors/OL6373539A", "/authors/OL7507879A", "/authors/OL485083A"].map(key => `key=${key.split('/')[2]}`).join('&')
```


/authors/merge?key=OL619897A&key=OL2908580A&key=OL5551708A&key=OL6373539A&key=OL7507879A&key=OL485083A
The hypothesis is that [web.seeother()](https://github.com/webpy/webpy/blob/master/web/webapi.py#L156-L163) is having troubles with URLs that contain Unicode characters.

Resolving patch:
diff --git a/openlibrary/plugins/upstream/merge_authors.py b/openlibrary/plugins/upstream/merge_authors.py
--- a/openlibrary/plugins/upstream/merge_authors.py
+++ b/openlibrary/plugins/upstream/merge_authors.py
@@ -298,8 +298,7 @@ def POST(self):
             return render_template("merge/authors", keys, top_books_from_author=top_books_from_author, formdata=formdata)
         else:
             # redirect to the master. The master will display a progressbar and call the merge_authors_json to trigger the merge.
-            master = web.ctx.site.get("/authors/" + i.master)
-            raise web.seeother(master.url() + "?merge=true&duplicates=" + ",".join(selected))
+            raise web.seeother("/authors/" + i.master + "/-/" + "?merge=true&duplicates=" + ",".join(selected))
 
 
 class merge_authors_json(delegate.page):


Test patch: