Repository: acl-org/acl-anthology

Issue:
Correction: Diacritics missing from author name though present in PDF
Ironically, [Diacritics Restoration Using Neural Networks](https://aclweb.org/anthology/papers/L/L18/L18-1247/) lists "Jan Hajic" on the page and in the BibTeX whereas it's spelled "Jan Hajič" in the PDF.

I see he is listed with the diacritic in some venues but not others, though going by the PDFs, Jan Hajič seems to be the preferred spelling.

Should the policy be that if an author is listed with multiple spellings differing only in diacritics, the one with the most diacritics should be applied?


Discussion:
The goal is generally for (a) BibTeX to reflect PDF and (b) author pages to collect all observed name variants. So this is a mistake that should be corrected in the XML.

Want to open a PR (and be added to our [list of volunteers](https://aclweb.org/anthology/info/credits/))?
Before making a one-time change I'd like to understand the underlying problem. Could it be that his name is listed without the diacritic in START, so it is showing up that way in the metadata for many of the venues?
I just checked -- indeed, his name in START is just `Jan Hajic`.
Missing diacritics is a widespread problem that we hoped to sidestep by allowing name variants. I suppose one could try to write a scraper to try to detect them.

If we knew when the switch was made to using START names, then looking for frequent mismatches after that year would help to identify people to contact and ask them to consider updating their profile.
I adapted the `auto_first_names.py` script and am running it on L18 now. It's catching quite a few errors; not just the one @nschneid pointed out, but removing _extra_ accents, decapitalizing an all-caps name, and flagging (but not autocorrecting, alas) a couple of misspelled names.
In L18 (528 papers, wow), the script made 150 changes (also wow) and printed another 100+ warnings that usually indicate a typo or missing word.

The automatic changes are easy to check, and they all look good except for a few:

```
INFO:L18-1066 author Tomasz Pędzimąż: changing: Pędzimąż -> Pȩdzima̧ż
INFO:L18-1495 author Anna Björk Nikulásdóttir: changing: Nikulásdóttir -> Nikulasdóttir
INFO:L18-1632 author Huda Almuzaini: changing: Almuzaini -> almuzaini
```

The first changes ogoneks to cedillas, I believe incorrectly.
The second one looks incorrect to me based on a Google search.
The third one lowercases the last name of someone who doesn't appear to do that regularly.

@mjpost, do you think the PDF should be followed in such cases?



Resolving patch:
diff --git a/bin/auto_accents.py b/bin/auto_accents.py
new file mode 100644
--- /dev/null
+++ b/bin/auto_accents.py
@@ -0,0 +1,153 @@
+"""
+Try to automatically restore accents in author names by downloading and scraping the PDFs.
+Reads and writes Anthology XML files.
+"""
+
+import tika.parser
+import requests
+import sys
+import lxml.etree as etree
+import re
+import unicodedata
+import os.path
+import yaml, yamlfix
+import copy
+
+def guess_url(paper):
+    url = paper.find('url') is not None and paper.find('url').text
+    if not url: url = paper.find('href') and paper.find('href').text
+    if not url: url = paper.attrib.get('href', None)
+    if not url:
+        volume = paper.getparent()
+        url = "http://www.aclweb.org/anthology/{}-{}".format(volume.attrib['id'], paper.attrib['id'])
+    return url
+
+def remove_accents(s):
+    s = unicodedata.normalize('NFKD', s)
+    s = ''.join(c for c in s if unicodedata.category(c) != 'Mn')
+    return s.lower()
+
+def scrape_authors(url):
+    logger.info("getting {}".format(url))
+    try:
+        r = requests.get(url)
+        if not r:
+            logger.error("could not download PDF")
+            return None
+        raw = tika.parser.from_buffer(r.content)
+        text = raw['content']
+    except KeyboardInterrupt:
+        raise
+    except Exception as e:
+        logger.error(str(e))
+        return {}
+    
+    index = {}
+    n = 0
+    for line in text.splitlines():
+        nospace = ''.join(line.split())
+        if nospace == 'Abstract': break
+        if nospace == '': continue
+
+        n += 1
+        if n > 50: break
+
+        # In PDF, sometimes i with accent is dotless i as in TeX,
+        # but NFKC doesn't get this
+        line = list(line)
+        for i in range(len(line)-1):
+            # bug: we should only be looking for accents above, not
+            # below
+            if line[i] == 'ı' and unicodedata.category(line[i+1]) == 'Mn':
+                line[i] = 'i'
+        line = ''.join(line)
+
+        # Remove email addresses, which are often same as names.
+        # There can be spaces inside the curly braces.
+        line = re.sub(r'(\{.*?\}|\S+)@\S+', '', line)
+
+        line = unicodedata.normalize('NFKC', line)
+        logger.debug('> '+line)
+        for word in line.split():
+            while len(word) > 0 and not (word[-1].isalpha() or
+                                         word[-1] in '.’'):
+                word = word[:-1]
+            rword = remove_accents(word)
+            if rword in index and index[rword] != word:
+                logging.warning("word '{}' appears twice with different accents".format(rword))
+                continue
+            index[rword] = word
+    #print(index)
+    return index
+
+def change(index, word):
+    rword = remove_accents(word)
+    if rword in index:
+        newword = index[rword]
+        if newword != word:
+            logger.info("changing: {} -> {}".format(word, newword))
+        return newword
+    else:
+        logger.warning("word {} in XML but not in PDF".format(rword))
+        return word
+
+if __name__ == "__main__":
+    import argparse
+    import logging
+    
+    # Set up logging
+    logger = logging.getLogger("auto_accents")
+    handler = logging.StreamHandler()
+    handler.setFormatter(logging.Formatter('%(levelname)s:%(location)s %(message)s'))
+    location = ""
+    def filter(r):
+        r.location = location
+        return True
+    handler.addFilter(filter)
+    #logger.setLevel(logging.INFO)
+    logger.setLevel(logging.DEBUG)
+    logger.addHandler(handler)
+    logger.propagate = False
+    
+    ap = argparse.ArgumentParser(description='Try to automatically restore accents in author names.')
+    ap.add_argument('infile', help="input XML file")
+    ap.add_argument('outfile', help="output XML file")
+    args = ap.parse_args()
+
+    scriptdir = os.path.dirname(os.path.abspath(__file__))
+
+    variants = yaml.load(open(os.path.join(scriptdir, '..', 'data', 'yaml', 'name_variants.yaml')))
+    variant_index = set()
+    for person in variants:
+        for name in [person['canonical']] + person.get('variants', []):
+            variant_index.add((name['first'], name['last']))
+
+    tree = etree.parse(args.infile)
+    volume = tree.getroot()
+    if not volume.tail: volume.tail = "\n"
+    for paper in volume.findall('paper'):
+        if paper.attrib['id'].endswith('000') or volume.attrib['id'].startswith('W') and paper.attrib['id'].endswith('00'): continue
+        paperid = '{}-{}'.format(volume.attrib['id'], paper.attrib['id'])
+        location = paperid + ":"
+
+        url = guess_url(paper)
+        index = scrape_authors(url)
+        if index is None: continue
+            
+        for xauthornode in paper.xpath('./author|./editor'):
+            xfirstnode = xauthornode.find('first')
+            xlastnode = xauthornode.find('last')
+            assert(len(xfirstnode) == len(xlastnode) == 0)
+
+            xfirst = (xfirstnode.text or "").strip()
+            xlast = xlastnode.text.strip()
+            location = '{} {} {} {}:'.format(paperid, xauthornode.tag, xfirst, xlast)
+
+            newfirst = ' '.join(change(index, word) for word in xfirst.split())
+            newlast = ' '.join(change(index, word) for word in xlast.split())
+            if (xfirst, xlast) in variant_index and (newfirst, newlast) not in variant_index:
+                logging.warning("add new name '{} {}' to name_variants.yaml".format(newfirst, newlast))
+            xfirstnode.text = newfirst
+            xlastnode.text = newlast
+                    
+    tree.write(args.outfile, xml_declaration=True, encoding='UTF-8')
diff --git a/bin/normalize_anth.py b/bin/normalize_anth.py
--- a/bin/normalize_anth.py
+++ b/bin/normalize_anth.py
@@ -86,6 +86,7 @@ def curly_quotes(s):
 
 def clean_unicode(s):
     s = s.replace('\u00ad', '') # soft hyphen
+    s = s.replace('\u2010', '-') # hyphen
 
     # Some sources encode an i with an accent above using dotless i,
     # which must be converted to normal i


Test patch: