Repository: acl-org/acl-anthology

Issue:
L18 URLs are wrong
Use this template to request a correction to the ACL Anthology.

Please ensure that you:

- [x] ...have read and understood the [criteria for corrections](http://aclanthology.info/corrections).
- [x] ...have included (if relevant) the ACL Anthology ID (e.g., J93-2003).

If approved, we will follow up with information about how to submit the correction.



Discussion:
In that, they point to Anthology PDFs instead of LREC-hosted ones?
I took a guess and looked in `L18.xml`. The contents of the (currently unused) `<url>` fields there indeed look wrong.
Exactly -- sorry I was so vague. It should be easy to fix, but I was in the middle of something else and just made this issue for later.

No need to apologize, tag-and-fill-later is a good MO. The issue is clear from a look at the XML.
It's the same with `L10`,`L12`,`L14`. However, the url of `paper` tag's attribute `href` is reachable.
What format does the LREC metadata come to us in? Do they use EasyChair?
Hi all:

Sorry to not clarify earlier on the L??.xml format.

From the legacy Anthology, we needed to distinguish URLs that the Anthology
serves as well as the original URL for materials where the 3rd party sister
organization gives us the rights to list their materials, but may not give
us the right to host their content.

<url/> = denotes the Anthology served address for the paper
<paper href=""/>, (only used LREC) href denotes the location on the third
party site for the canonical paper.
<href/>, (only used in TACL) href denotes the location on the ACL
affiliated site for the canonical paper.

Probably we should just unify the use of <href/> and <paper href=""/> to
the former.

During my time, we had many fewer workers on the Anthology and had to
marginalize attention to sister organizations in terms of priority on
ingestions and corrections, and LREC was one of those conferences that was
affected.

ELRA recently changed their stance to allow the ACL Anthology to host their
content as well, so now they join all of the other current sister
organizations that give and indemnify ACL from listing and redistributing
their proceedings.  The copyright still belongs to the sister organization,
so any requests for reprinting and repurposing must go to their executive
committees.

Anyways, to get the the point: LREC provides no official metadata to ACL,
however Vladimir Popescu from ELRA compiled XML metadata for L18 and
provided it to me, which I have forwarded to Matt.  For L14 and earlier, I
harvested the metadata directly from the crawl of the website.  L16 is the
gap left to address.


Resolving patch:
diff --git a/bin/repair_url.py b/bin/repair_url.py
new file mode 100644
--- /dev/null
+++ b/bin/repair_url.py
@@ -0,0 +1,60 @@
+"""Repairs URLs in Anthology XML files.
+
+Usage: repair_url.py <infilename> <outfilename>
+
+To do:
+- check URLs of attachments
+- incorporate into a more general XML-fixing script
+
+"""
+
+import lxml.etree as etree
+import requests
+import sys
+
+def test_url(url):
+    #sys.stderr.write("retrieving {}: ".format(url))
+    r = requests.head(url, allow_redirects=True)
+    #sys.stderr.write("{}\n".format(r.status_code))
+    return r.status_code == requests.codes.ok
+
+filename = sys.argv[1]
+outfilename = sys.argv[2]
+tree = etree.parse(filename)
+volume = tree.getroot()
+for paper in volume.findall("paper"):
+    if 'href' in paper.attrib:
+        if not test_url(paper.attrib['href']):
+            sys.stderr.write('{}:{} removing href attribute: {}\n'.format(filename, paper.sourceline, paper.attrib['href']))
+            del paper.attrib['href']
+
+    href = paper.find('href')
+    if href is not None:
+        assert len(href) == 0
+        if not test_url(href.text):
+            sys.stderr.write('{}:{} removing href element: {}\n'.format(filename, href.sourceline, href.text))
+            paper.remove(href)
+    
+    anth_url = 'http://www.aclweb.org/anthology/{}-{:04d}'.format(volume.attrib['id'], int(paper.attrib['id']))
+    anth_url_good = test_url(anth_url)
+    
+    url = paper.find("url")
+    assert url is None or len(url) == 0
+    
+    if url is None:
+        if anth_url_good:
+            url = etree.Element('url')
+            url.text = anth_url
+            sys.stderr.write("{}:{} inserting url element: {}\n".format(filename, paper.sourceline, anth_url))
+            paper.append(url)
+        
+    else:
+        if anth_url_good and url.text != anth_url:
+            sys.stderr.write("{}:{} rewriting url: {} -> {}\n".format(filename, url.sourceline, url.text, anth_url))
+            url.text = anth_url
+
+        else:
+            sys.stderr.write("{}:{} removing url element because {} is bad\n".format(filename, url.sourceline, anth_url))
+            paper.remove(url)
+    
+tree.write(outfilename, encoding='UTF-8', xml_declaration=True, with_tail=True)


Test patch: