Repository: acl-org/acl-anthology

Issue:
LREC full author names
(Original message replaced) L06 only uses first initials, even though the original papers have full names. This makes author indexing difficult. Unfortunately, the online proceedings (http://www.lrec-conf.org/proceedings/lrec2006/) are the same. Is there a way to get the full names other than manually entering them? Scraping them from the PDFs?


Discussion:
Other meetings have the same issue, but L06 is by far the biggest.

```
1665 L06.xml
 291 H01.xml
  86 L04.xml
  81 W04.xml
  79 M98.xml
  75 L02.xml
  75 H90.xml
  69 L00.xml
  60 H91.xml
  54 C82.xml
```
We could talk to the LREC folks. I'm not sure what system they use for conference management, but they might have the rest of the information available. I'll send an email.
For LREC, at least, it looks like it was specific to that year. PDF scraping seems possible but also probably a research problem. I wonder if we could get funding for this, e.g., https://www.imls.gov/grants/available/national-leadership-grants-libraries.

Note that LREC has all the abstracts on their paper pages, which we don't have ingested. We could add them and it would probably help with indexing.
Just found this: Semantic Scholar somehow has full author names, e.g., https://www.semanticscholar.org/paper/A-Cross-language-Approach-to-Rapid-Creation-of-New-Feldman-Hana/787bfd0ee5b5ccd891cf33bc84166d3a02a7b640

(They offered help, see note here: https://github.com/acl-org/acl-anthology/issues/208#issuecomment-477312810)

Resolving patch:
diff --git a/bin/auto_first_names.py b/bin/auto_first_names.py
new file mode 100644
--- /dev/null
+++ b/bin/auto_first_names.py
@@ -0,0 +1,97 @@
+"""
+Try to automatically fill in author first names by downloading and scraping the PDFs.
+Reads and writes Anthology XML files.
+
+Bugs:
+
+- If two authors on the same paper have same last name, one of them will get the wrong 
+  first name. This could be remedied somewhat by using the first initial as a clue. 
+  Otherwise, we should check for duplicate names.
+- If name is broken across two lines, there is no first name.
+- Strip punctuation from XML names too
+- Strip "and" from beginning
+"""
+
+import tika.parser
+import requests
+import sys
+import lxml.etree as etree
+import re
+import unicodedata
+import os.path
+
+if len(sys.argv) != 3:
+    sys.exit("usage: auto_first_names.py <input-xml> <output-xml>")
+
+tree = etree.parse(sys.argv[1])
+for paper in tree.findall('paper'):
+    print("paper", paper.attrib['id'])
+    url = paper.find('url') and paper.find('url').text
+    if url is None: url = paper.find('href') and paper.find('href').text
+    if url is None: url = paper.attrib.get('href', None)
+    if url is None:
+        filename = os.path.basename(sys.argv[1])
+        assert filename.endswith('.xml')
+        filename = filename[:-4]
+        url = "http://www.aclweb.org/anthology/{}-{}".format(filename, paper.attrib['id'])
+    if url is None:
+        print('no url found; skipping')
+        print()
+        continue
+
+    print("getting", url)
+    try:
+        req = requests.get(url)
+        raw = tika.parser.from_buffer(req.content)
+        text = [line for line in raw['content'].splitlines() if line.strip() != ""]
+    except KeyboardInterrupt:
+        raise
+    except Exception as e:
+        print("error: {}".format(e))
+        continue
+
+    index = {}
+    for line in text[:20]:
+        if line.strip() == 'Abstract': break
+        line = unicodedata.normalize('NFKC', line)
+        print('> '+line)
+        names = re.split(r',\s*|\s+and\s+|,\s*and\s+|&', line)
+        for name in names:
+            # strip leading and trailing numbers, punctuation, symbols, whitespace
+            # bugs: can a name end in an apostrophe?
+            while len(name) > 0 and unicodedata.category(name[0])[0] in "NPSZ":
+                name = name[1:]
+            while len(name) > 0 and unicodedata.category(name[-1])[0] in "NPSZ":
+                name = name[:-1]
+            for part in name.split():
+                if part not in index: # ignore subsequent mentions, which may be from text
+                    index[part.lower()] = name
+
+    allnames = set()
+    for xauthornode in paper.findall('author'):
+        xfirstnode = xauthornode.find('first')
+        xlastnode = xauthornode.find('last')
+        assert(len(xfirstnode) == len(xlastnode) == 0)
+        
+        xfirst = xfirstnode.text.strip()
+        xlast = xlastnode.text.strip()
+        xname = '{} {}'.format(xfirst, xlast) # just used for logging
+        
+        if xlast.lower() not in index:
+            print("warning: {} not found; skipping".format(xname))
+            continue
+        
+        pname = index[xlast.lower()]
+        i = pname.lower().find(xlast.lower())
+        newfirst = pname[:i].strip()
+        afterlast = pname[i+len(xlast):].strip()
+        if afterlast:
+            print("warning: {}: trailing string after last name: {}".format(xname, afterlast))
+        if (newfirst, xlast) in allnames:
+            print("warning: {}: duplicate new first name: {}".format(xname, newfirst))
+        allnames.add((newfirst, xlast))
+        print("{} {} -> {} {}".format(xfirst, xlast, newfirst, xlast))
+        xfirstnode.text = newfirst
+
+    print()
+tree.write(sys.argv[2], xml_declaration=True, encoding='UTF-8', with_tail=True)


Test patch: