Repository: hydroshare/hydroshare

Issue:
test_resourcemeta_sax_parsing fails a lot in CI testing
The [line](https://github.com/hydroshare/hydroshare/blob/develop/hs_core/tests/serialization/test_resourcemeta_sax_parsing.py#L118) is flagged frequently in CI (PR or commit builds) and generates the following error.

**Error Message**

```
[Errno socket error] [Errno 110] Connection timed out
```

**Stack trace**

```
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/docker/hydroshare/hs_core/tests/serialization/test_resourcemeta_sax_parsing.py", line 118, in test_sax_parsing
    xml.sax.parseString(self.parse_sample, handler)
  File "/usr/local/lib/python2.7/xml/sax/__init__.py", line 49, in parseString
    parser.parse(inpsrc)
  File "/usr/local/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/local/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/local/lib/python2.7/xml/sax/expatreader.py", line 397, in external_entity_ref
    "")
  File "/usr/local/lib/python2.7/xml/sax/saxutils.py", line 349, in prepare_input_source
    f = urllib.urlopen(source.getSystemId())
  File "/usr/local/lib/python2.7/urllib.py", line 87, in urlopen
    return opener.open(url)
  File "/usr/local/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/local/lib/python2.7/urllib.py", line 350, in open_http
    h.endheaders(data)
  File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/local/lib/python2.7/socket.py", line 575, in create_connection
    raise err
IOError: [Errno socket error] [Errno 110] Connection timed out
```

@pkdash, @hyi, @jbradberry - It does not always fail, but fails often enough to warrant a look.



Discussion:
Perhaps relevant as pointed out by @jbradberry in hip chat: [https://stuartsierra.com/2008/05/08/stop-your-java-sax-parser-from-downloading-dtds](https://stuartsierra.com/2008/05/08/stop-your-java-sax-parser-from-downloading-dtds)

https://docs.python.org/2/library/xml.sax.reader.html#xml.sax.xmlreader.XMLReader.setEntityResolver
https://docs.python.org/2/library/xml.sax.handler.html#xml.sax.handler.EntityResolver

@jbradberry @mjstealey Has this issue been fixed? I remember seeing a PR for it. 

@hyi I have a fix for it that I produced on Friday, but haven't produced the PR for it yet.  Expect it later today.  cc @mjstealey 


Resolving patch:
diff --git a/hs_core/xmlparser.py b/hs_core/xmlparser.py
new file mode 100644
--- /dev/null
+++ b/hs_core/xmlparser.py
@@ -0,0 +1,61 @@
+import base64
+import os.path
+from StringIO import StringIO
+from xml.sax import xmlreader, saxutils
+from xml.sax.expatreader import ExpatParser
+
+from django.conf import settings
+
+
+XML_CACHE = os.path.join(settings.PROJECT_ROOT, '_xmlcache')
+
+
+class CatalogedExpatParser(ExpatParser):
+    def external_entity_ref(self, context, base, sysid, pubid):
+        if not self._external_ges:
+            return 1
+
+        source = self._ent_handler.resolveEntity(pubid, sysid)
+        source = saxutils.prepare_input_source(source,
+                                               self._source.getSystemId() or
+                                               "")
+
+        # If an entry does not exist in the xml cache, create it.
+        filepath = os.path.join(XML_CACHE, base64.urlsafe_b64encode(pubid))
+        if not os.path.isfile(filepath):
+            with open(filepath, 'w') as f:
+                contents = source.getByteStream().read()
+                source.setByteStream(StringIO(contents))
+                f.write(contents)
+
+        self._entity_stack.append((self._parser, self._source))
+        self._parser = self._parser.ExternalEntityParserCreate(context)
+        self._source = source
+
+        try:
+            xmlreader.IncrementalParser.parse(self, source)
+        except:
+            return 0  # FIXME: save error info here?
+
+        (self._parser, self._source) = self._entity_stack[-1]
+        del self._entity_stack[-1]
+        return 1
+
+
+class CatalogEntityResolver(object):
+    def resolveEntity(self, publicId, systemId):
+        filepath = os.path.join(XML_CACHE, base64.urlsafe_b64encode(publicId))
+        if os.path.isfile(filepath):
+            source = xmlreader.InputSource()
+            with open(filepath, 'r') as f:
+                source.setByteStream(StringIO(f.read()))
+            return source
+
+        return systemId
+
+
+def create_parser(*args, **kwargs):
+    parser = CatalogedExpatParser(*args, **kwargs)
+    parser.setEntityResolver(CatalogEntityResolver())
+
+    return parser


Test patch: