Repository: obspy/obspy

Issue:
Invalid resource identifier for QuakeML produced by SC3
This is a follow-up of the [discussion](https://lists.swapbytes.de/archives/obspy-users/2018-March/002702.html) in the mailing list, which I report here:

The attached QuakeML file ([ipgp2016afqtcu.xml.gz](https://github.com/obspy/obspy/files/1827019/ipgp2016afqtcu.xml.gz)) has been produced by SeisComP3.

It is correctly read by `read_events()`, but when trying to write it back to disk, I get the following error:

```python
ValueError: The id 'smi:org.gfz-potsdam.de/geofon/RMHP(60)>>ITAPER(3)>>BW(4,5,15)' is not a valid QuakeML resource identifier.
```

This is related to the presence of the sign `>` which is encoded in the original xml file as `&gt;` but gets decoded as `>`.

This symbol is part of the SeisComP3 filter grammar: https://www.seiscomp3.org/doc/jakarta/2015.040/base/filter-grammar.html

I think that, as long as this symbol is not decoded from its HTML representation, it should be acceptable. 

So my impression is that this is an ObsPy bug. 


Discussion:
The same file is not valid according to `jing`, but due to a different error:

```
ipgp2016afqtcu.xml:61:127: error: value of attribute "publicID" is invalid; must be a URI
```

That's because an URI cannot have more than one `#` sign.

By manually fixing this problem (see [ipgp2016afqtcu_fix.xml.gz](https://github.com/obspy/obspy/files/1827033/ipgp2016afqtcu_fix.xml.gz)), the resulting file is valid XML, according to `jing`, even with those `&gt;` in it.
This has been discussed in the mailing list with @krischer, but I'm not sure whether it is an ObsPy bug or a SeisComP3 bug, so I call here @andres-h and @gempa-jabe as well.
Thanks!

Edit: also ping @jmsaurel 
See also #1872
@andres-h @gempa-jabe pinging you again so we can discuss this. Its getting a bit lengthy on the mailing list.

The question boils down whether or not resource ids like `smi:org.gfz-potsdam.de/geofon/RMHP(60)>>ITAPER(3)>>BW(4,5,15)` are valid in QuakeML or not. In the raw XML it is encoded as `smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,15)`.

Validation tools don't complain but I personally think that the regular expression for the allowable characters has to be applied at the decoded representation which would mean that these ids are not valid.
Pinging also @feuchner and @kaestli in case they want/can weight in.
I think a clean solution would be using filter ID without any HTML special characters, but at the end it's @gempa-jabe who decides.

In any case, there will be QuakeML generated with old SC3 versions, so it would be good if this error can be changed to a warning at least.
Using filterID without any XML specific characters would help in this particular case but it is unfortunately not a generic solution. We need to find a way to safely transform any string to an smi and back. I don't have a perfect solution to that right now. We also can't change the filters as we are using them in SC3, so the simplest approach is to ignore filterID while converting to QuakeML. But that does not prevent anyone from configuring invalid publicIDs that do not fit into the smi scheme. I could imagine to encode the IDs as base64 but there is no notion that this has been done, so the way back is not clear. I agree with @andres-h that this won't help with documents created with older SC3 version. Relaxing the smi restrictions or issue a warning would help here.
Thanks @andres-h and @gempa-jabe for the replies! I will continue the discussion on #2093 on how to implement a fix.

Resolving patch:
diff --git a/obspy/core/event/base.py b/obspy/core/event/base.py
--- a/obspy/core/event/base.py
+++ b/obspy/core/event/base.py
@@ -780,7 +780,7 @@ def get_quakeml_uri(self, authority_id="local"):
             id = str(uuid4())
 
         regex = r"^(smi|quakeml):[\w\d][\w\d\-\.\*\(\)_~']{2,}/[\w\d\-\." + \
-                r"\*\(\)_~'][\w\d\-\.\*\(\)\+\?_~'=,;#/&amp;]*$"
+                r"\*\(\)_~'][\w\d\-\.\*\(\)\+\?_~'=,;#/&]*$"
         result = re.match(regex, str(id))
         if result is not None:
             return id
diff --git a/obspy/io/pde/mchedr.py b/obspy/io/pde/mchedr.py
--- a/obspy/io/pde/mchedr.py
+++ b/obspy/io/pde/mchedr.py
@@ -1012,7 +1012,9 @@ def _parse_record_s(self, line, event, p_pick, p_arrival):
 
     def _deserialize(self):
         catalog = Catalog()
-        res_id = '/'.join((res_id_prefix, self.filename))
+        res_id = '/'.join((res_id_prefix,
+                           self.filename.replace(':', '/')))\
+            .replace('\\', '/').replace('//', '/')
         catalog.resource_id = ResourceIdentifier(id=res_id)
         catalog.description = 'Created from NEIC PDE mchedr format'
         catalog.comments = ''
diff --git a/obspy/io/quakeml/core.py b/obspy/io/quakeml/core.py
--- a/obspy/io/quakeml/core.py
+++ b/obspy/io/quakeml/core.py
@@ -1082,11 +1082,15 @@ def _id(self, obj):
         try:
             return obj.get_quakeml_uri()
         except Exception:
-            return ResourceIdentifier().get_quakeml_uri()
+            msg = ("'%s' is not a valid QuakeML URI. It will be in the final "
+                   "file but note that the file will not be a valid QuakeML "
+                   "file.")
+            warnings.warn(msg % obj.id)
+            return obj.id
 
     def _str(self, value, root, tag, always_create=False, attrib=None):
         if isinstance(value, ResourceIdentifier):
-            value = value.get_quakeml_uri()
+            value = self._id(value)
         if always_create is False and value is None:
             return
         etree.SubElement(root, tag, attrib=attrib).text = "%s" % value


Test patch:
diff --git a/obspy/io/quakeml/tests/data/invalid_id.xml b/obspy/io/quakeml/tests/data/invalid_id.xml
new file mode 100644
--- /dev/null
+++ b/obspy/io/quakeml/tests/data/invalid_id.xml
@@ -0,0 +1,91 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<q:quakeml xmlns="http://quakeml.org/xmlns/bed/1.2"
+  xmlns:q="http://quakeml.org/xmlns/quakeml/1.2">
+  <eventParameters publicID="smi:eu.emsc/unid/123456">
+    <event publicID="smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,15)">
+      <magnitude publicID="smi:ch.ethz.sed/magnitude/37465">
+        <mag>
+          <value>5.5</value>
+          <uncertainty>0.1</uncertainty>
+        </mag>
+        <type>MS</type>
+        <originID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,26)</originID>
+        <methodID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,27)</methodID>
+      </magnitude>
+      <pick publicID="smi:org.gfz-potsdam.de/geofon/Pick#20160627194457.143264.18834">
+        <phaseHint>P</phaseHint>
+        <evaluationMode>manual</evaluationMode>
+        <creationInfo>
+          <agencyID>OVSM</agencyID>
+          <author>clouard</author>
+          <creationTime>2016-06-27T19:44:57.14344Z</creationTime>
+        </creationInfo>
+        <time>
+          <value>2016-01-04T01:48:01.560346Z</value>
+          <lowerUncertainty>0.05000000075</lowerUncertainty>
+          <upperUncertainty>0.05000000075</upperUncertainty>
+        </time>
+        <waveformID networkCode="MQ" stationCode="CPM" locationCode="91" channelCode="EHZ"/>
+        <filterID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,16)</filterID>
+        <methodID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,28)</methodID>
+        <slownessMethodID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,29)</slownessMethodID>
+      </pick>
+      <amplitude publicID="smi:nz.org.geonet/event/2806038g/amplitude/1/modified">
+        <genericAmplitude>
+          <value>1.0e-08</value>
+        </genericAmplitude>
+        <filterID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,20)</filterID>
+        <methodID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,17)</methodID>
+        <pickID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,18)</pickID>
+        <creationInfo>
+          <agencyID>EMSC</agencyID>
+          <agencyURI>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,30)</agencyURI>
+          <author>Erika Mustermann</author>
+          <authorURI>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,31)</authorURI>
+          <creationTime>2012-04-04T16:40:50.000000Z</creationTime>
+          <version>1.0.1</version>
+        </creationInfo>
+      </amplitude>
+      <stationMagnitude publicID="smi:ch.ethz.sed/magnitude/station/881342">
+        <originID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,21)</originID>
+        <amplitudeID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,22)</amplitudeID>
+        <methodID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,23)</methodID>
+        <mag>
+          <value>6.5</value>
+          <uncertainty>0.2</uncertainty>
+        </mag>
+      </stationMagnitude>
+      <origin publicID="smi:www.iris.edu/ws/event/query?originId=7680412">
+        <time>
+          <value>2011-03-11T05:46:24.120000Z</value>
+        </time>
+        <latitude>
+          <value>38.297</value>
+        </latitude>
+        <longitude>
+          <value>142.373</value>
+        </longitude>
+        <arrival publicID="smi:some/arrival/id">
+          <pickID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,32)</pickID>
+          <phase>Pn</phase>
+          <azimuth>12.0</azimuth>
+          <distance>0.5</distance>
+          <takeoffAngle>
+            <value>11.0</value>
+            <uncertainty>0.2</uncertainty>
+          </takeoffAngle>
+          <timeResidual>1.6</timeResidual>
+          <horizontalSlownessResidual>1.7</horizontalSlownessResidual>
+          <backazimuthResidual>1.8</backazimuthResidual>
+          <timeWeight>0.48</timeWeight>
+          <horizontalSlownessWeight>0.49</horizontalSlownessWeight>
+          <backazimuthWeight>0.5</backazimuthWeight>
+          <earthModelID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,33)</earthModelID>
+        </arrival>
+        <referenceSystemID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,34)</referenceSystemID>
+        <methodID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,35)</methodID>
+        <earthModelID>smi:org.gfz-potsdam.de/geofon/RMHP(60)&gt;&gt;ITAPER(3)&gt;&gt;BW(4,5,36)</earthModelID>
+      </origin>
+    </event>
+  </eventParameters>
+</q:quakeml>
diff --git a/obspy/io/quakeml/tests/data/neries_events.xml b/obspy/io/quakeml/tests/data/neries_events.xml
--- a/obspy/io/quakeml/tests/data/neries_events.xml
+++ b/obspy/io/quakeml/tests/data/neries_events.xml
@@ -1,6 +1,6 @@
 <?xml version='1.0' encoding='UTF-8'?>
 <quakeml xmlns="http://quakeml.org/xmlns/quakeml/1.0">
-  <eventParameters publicID="smi://eu.emsc/unid">
+  <eventParameters publicID="smi:eu.emsc/unid">
     <event publicID="quakeml:eu.emsc/event/20120404_0000041">
       <preferredOriginID>quakeml:eu.emsc/origin/rts/261020/782484</preferredOriginID>
       <preferredMagnitudeID>quakeml:eu.emsc/NetworkMagnitude/rts/261020/782484/796646</preferredMagnitudeID>
diff --git a/obspy/io/quakeml/tests/test_quakeml.py b/obspy/io/quakeml/tests/test_quakeml.py
--- a/obspy/io/quakeml/tests/test_quakeml.py
+++ b/obspy/io/quakeml/tests/test_quakeml.py
@@ -1062,6 +1062,32 @@ def test_avoid_empty_stub_elements(self):
         self.assertEqual(cat[0].focal_mechanisms[0].nodal_planes, None)
         self.assertEqual(cat[0].focal_mechanisms[0].principal_axes, None)
 
+    def test_writing_invalid_quakeml_id(self):
+        """
+        Some ids might be invalid. We still want to write them to not mess
+        with any external tools relying on the ids. But we also raise a
+        warning of course.
+        """
+        filename = os.path.join(self.path, 'invalid_id.xml')
+        cat = read_events(filename)
+        self.assertEqual(
+            cat[0].resource_id.id,
+            "smi:org.gfz-potsdam.de/geofon/RMHP(60)>>ITAPER(3)>>BW(4,5,15)")
+        with NamedTemporaryFile() as tf:
+            with warnings.catch_warnings(record=True) as w:
+                warnings.simplefilter("always")
+                cat.write(tf.name, format="quakeml")
+                cat2 = read_events(tf.name)
+        self.assertEqual(len(w), 19)
+        self.assertEqual(
+            w[0].message.args[0],
+            "'smi:org.gfz-potsdam.de/geofon/RMHP(60)>>ITAPER(3)>>BW(4,5,15)' "
+            "is not a valid QuakeML URI. It will be in the final file but "
+            "note that the file will not be a valid QuakeML file.")
+        self.assertEqual(
+            cat2[0].resource_id.id,
+            "smi:org.gfz-potsdam.de/geofon/RMHP(60)>>ITAPER(3)>>BW(4,5,15)")
+
 
 def suite():
     return unittest.makeSuite(QuakeMLTestCase, 'test')