Repository: acl-org/acl-anthology

Issue:
LaTeX processing is not being done on ingestion
This came up in #628, and I think it appeared for the first time for EMNLP 2019 because we pushed some simplifying changes to START that turned off LaTeX on their end. 

Is it because `normalize_anth.py` is not being run with the `-t` option?

The problem is that currently, `normalize_anth.py` cannot be rerun with the `-t` option; all kinds of errors come up. It ought to be fixable but might not be an easy fix.


Discussion:
> Is it because normalize_anth.py is not being run with the -t option?

Ah, yes, this must be it!

I do not call `normalize_anth.py` directly, but only call it via `bin/ingest.py`, where I call 'process()', but pass it "xml" instead of "latex".

Can I use `normalize_anth.py` while reading in an XML file?
Sorry, I didn't understand the last question...
It looks like I should change [this line](https://github.com/acl-org/acl-anthology/blob/20d6c25c08d0f7ee8b1eaceae6aea67556eeea80/bin/ingest.py#L87), passing "latex" instead of "xml". Is that correct?

[Edit: added the link]
Right, with the caveat that LaTeX processing should not be done more than once.


Ingest is called just once, so this is the perfect place for it.

Resolving patch:
diff --git a/bin/ingest.py b/bin/ingest.py
--- a/bin/ingest.py
+++ b/bin/ingest.py
@@ -32,7 +32,7 @@
 
 from datetime import datetime
 
-from normalize_anth import process
+from normalize_anth import normalize
 from anthology.utils import make_nested, make_simple_element, build_anthology_id, indent
 from anthology.index import AnthologyIndex
 from anthology.people import PersonName
@@ -84,7 +84,7 @@
     # Normalize
     for paper in root_being_added.findall('.//paper'):
         for oldnode in paper:
-            process(oldnode, informat='xml')
+            normalize(oldnode, informat='latex')
 
     # Ingest each volume.
     # First, find the XML file.
diff --git a/bin/normalize_anth.py b/bin/normalize_anth.py
--- a/bin/normalize_anth.py
+++ b/bin/normalize_anth.py
@@ -121,14 +121,24 @@ def decompose(c):
 
     return s
 
-def process(oldnode, informat):
+
+def normalize(oldnode, informat):
+    """
+    Receives an XML 'paper' node and normalizes many of its fields, including:
+    - Unescaping HTML
+    - Normalizing quotes and other punctuation
+    - Mapping many characters to unicode
+    In addition, if the 'informat' is "latex", it will convert many LaTeX characters
+    to unicode equivalents. Note that these latter LaTeX operations are not idempotent.
+    """
+
     if oldnode.tag in ['url', 'href', 'mrf', 'doi', 'bibtype', 'bibkey',
                        'revision', 'erratum', 'attachment', 'paper',
                        'presentation', 'dataset', 'software', 'video']:
         return
     elif oldnode.tag in ['author', 'editor']:
         for oldchild in oldnode:
-            process(oldchild, informat=informat)
+            normalize(oldchild, informat=informat)
     else:
         if informat == "latex":
             if len(oldnode) > 0:
@@ -138,7 +148,7 @@ def process(oldnode, informat):
             newnode.tag = oldnode.tag
             newnode.attrib.update(oldnode.attrib)
             replace_node(oldnode, newnode)
-            
+
         maptext(oldnode, html.unescape)
         maptext(oldnode, curly_quotes)
         maptext(oldnode, clean_unicode)
@@ -170,6 +180,6 @@ def process(oldnode, informat):
                                                paper.attrib['id'])
         for oldnode in paper:
             location = "{}:{}".format(papernum, oldnode.tag)
-            process(oldnode, informat=informat)
-                
+            normalize(oldnode, informat=informat)
+
     tree.write(args.outfile, encoding="UTF-8", xml_declaration=True)


Test patch: