Repository: obspy/obspy

Issue:
read_inventory converts empty sub-elements to the 'None' string
When read_inventory() reads a stationxml file, it converts empty sub-elements to the 'None' string.

Looking at obspy/io/stationxml/core.py:

```python
710 def _read_equipment(equip_element, _ns):
713     description = _tag2obj(equip_element, _ns("Description"), str)
...
1448 def _tag2obj(element, tag, convert):
1449     # we use future.builtins.str and are sure we have unicode here
1450     try:
1451         return convert(element.find(tag).text)
```

When an element is present (e.g., sensor) but the subelement is not (e.g., sensor.description),
_tag2obj returns str(None) which is the string 'None'.

So if a stationxml containing:

    <Sensor>
         <Description></Description>
    </Sensor>

is read in and then written back out using inventory.write(), it writes:

    <Sensor>
        <Description>None</Description>
    </Sensor>

i.e., it writes 'None' in place of the empty sub-element.
Is this intentional ?  I know the xsd specifies that a sensor must have a description, but I've seen stationxml files out in the wild that are missing it.

I just don't know if it's better to keep the field empty on disk (=None in memory) or to use the "None" string as placeholder ?

Thanks!


Discussion:
`element.text` on `<Description></Description>` gives `None`? Really..? I'd really expect it to give `''`
You got code to reproduce? Because it should be fine, from what I see..

```
In [5]: elem = lxml.etree.Element('blap')

In [6]: elem
Out[6]: <Element blap at 0x7fe44d001a28>

In [7]: elem.text = ''

In [8]: lxml.etree.tostring(elem)
Out[8]: '<blap></blap>'

In [9]: elem.text
Out[9]: ''

In [10]: str(elem.text)
Out[10]: ''
```
Oh nevermind.. thats really unexpected and ugly

```python
In [13]: elem = lxml.etree.fromstring('<blap></blap>')

In [14]: str(elem.text)
Out[14]: 'None'
```
So, in terms of xml semantics.. that would imply that `<blap/>` and `<blap></blap>` are considered 100% equivalent?
I don't know what the right answer is on this.

On one hand, xml is complicated to handle and it would be easier to just say
that obspy won't handle any stationxml that doesn't adhere to the xsd.

On the other hand, as far as I know, the xsd just says that an element must
be present
e.g.,that Sensor must contain <Description></Description>
It doesn't say it can't be empty (unless there is a way to specify this ?)

So what to do ?

On Wed, Dec 18, 2019 at 10:32 AM Tobias Megies <notifications@github.com>
wrote:

> Oh nevermind.. thats really unexpected and ugly
>
> In [13]: elem = lxml.etree.fromstring('<blap></blap>')
>
> In [14]: str(elem.text)
> Out[14]: 'None'
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/obspy/obspy/issues/2519?email_source=notifications&email_token=AAQMBQRQ2Q5FUVQEOU5ZEVDQZI7ADA5CNFSM4J4MJY5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHGPX7I#issuecomment-567081981>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAQMBQSQFD66EKL3XST4GCTQZI7ADANCNFSM4J4MJY5A>
> .
>

@barsch what's your take on this? it seems there is actually discussion about whether it's the same or not..? https://stackoverflow.com/a/7232009
This seems reasonable to me:

Yes, it's not as simple as it seems at first.
XSD differences:

First of all, it depends on your data type. If you use an XSD schema
definition, which defines the types of your elements, then
<element></element> *can actually only be used for string types*. Because,
that's what it is, it is actually an empty string value: "".

Therefore, it is illegal to use <element></element> for an integer, while
by contrast <element/> is applicable for all simple types.

==============================================
So maybe for expected string fields, you should convert between None in
obspy and "" in stationxml (?)

Something like:

def _tag2obj(element, tag, convert):
    # we use future.builtins.str and are sure we have unicode here
    if isinstance(str, convert) and len(re.sub(r'\s+', '',
element.find(tag).text)) == 0:
        return None
    try:
        return convert(element.find(tag).text)
    except Exception:
        None

-M

On Wed, Dec 18, 2019 at 1:10 PM Tobias Megies <notifications@github.com>
wrote:

> @barsch <https://github.com/barsch> what's your take on this? it seems
> there is actually discussion about whether it's the same or not..?
> https://stackoverflow.com/a/7232009
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/obspy/obspy/issues/2519?email_source=notifications&email_token=AAQMBQS7GPFHVCDCFPW7KGTQZJRSXA5CNFSM4J4MJY5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHG7VXI#issuecomment-567147229>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAQMBQRY7SGXENDDW4VCOH3QZJRSXANCNFSM4J4MJY5A>
> .
>

> So maybe for expected string fields, you should convert between None in obspy and "" in stationxml (?)

The thing is.. we can't tell, that should be done in lxml. lxml passes on `None` when it actually should be `''`.

```
In [13]: elem = lxml.etree.fromstring('<blap></blap>')

In [14]: str(elem.text)
Out[14]: 'None'
```

But yes, it looks like that's the only thing we can do..
Tobias, you're right - it seems that if an element only has sub-elements,
then it has element.text = string of whitespace,
which becomes the empty string when you strip out the whitespace.
e.g., for elem = CalibrationUnits (below), elem.text = "    "  (whitespace
string)

So this: <Description></Description>  is the same as <Description/> and
evaluates to elem.text = None,
while:   <Description> </Description> evaluates to elem.text = ""

So you'd need to know that you have an inner element with no sub-elements -
e.g., Description, Name, etc. - and then
  if element.text is None or len(element.text.strip_whitespace()) == 0:
      return None

But you wouldn't do this for any element (e.g., CalibrationUnits) that
contains a sub-element.

And I guess you need to repeat the tests when writing out the xml:
    if channel.calibration_units:
        ...
        if channel.calibration_units_description:
            etree.SubElement(cu, "Description").text = \
                str(channel.calibration_units_description)        // I'm
assuming this would convert None to 'None'

So you'd have to test that the obspy object attributes were None and set
text="" empty string for write.

I can play around with this in obspy and make sure it works.

-Mike

foo = """
    <myroot>
        <CalibrationUnits>
            <Description> </Description>
        </CalibrationUnits>
        <CalibrationUnits>
            <Description></Description>
        </CalibrationUnits>
        <CalibrationUnits>
            <Description/>
        </CalibrationUnits>
    </myroot>"""

    root = lxml.etree.fromstring(foo)
    for tag in ['CalibrationUnits', 'Description']:
        for elem in root.iter(tag):
            if not getattr(elem, 'text', None):
                print("elem=%s text=[None]" % elem)
            else:
                text = strip_white(elem.text)
                print("elem=%s text=[%s] len=%s" % (elem, text, len(text)))

import re
def strip_white(string):
    return(re.sub(r'\s+', '', string))

elem=<Element CalibrationUnits at 0x1a0cef18c8> text=[] len=0
elem=<Element CalibrationUnits at 0x1a0cef1908> text=[] len=0
elem=<Element CalibrationUnits at 0x1a0cef1948> text=[] len=0
elem=<Element Description at 0x1a0cef1908> text=[] len=0
elem=<Element Description at 0x1a0cef18c8> text=[None]
elem=<Element Description at 0x1a0cef19c8> text=[None]


On Thu, Dec 19, 2019 at 4:32 AM Tobias Megies <notifications@github.com>
wrote:

> So maybe for expected string fields, you should convert between None in
> obspy and "" in stationxml (?)
>
> The thing is.. we can't tell, that should be done in lxml. lxml passes on
> None when it actually should be ''.
>
> In [13]: elem = lxml.etree.fromstring('<blap></blap>')
>
> In [14]: str(elem.text)
> Out[14]: 'None'
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/obspy/obspy/issues/2519?email_source=notifications&email_token=AAQMBQTRVSR5QXVU5AVQFQLQZM5T3A5CNFSM4J4MJY5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJAADA#issuecomment-567410700>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAQMBQTHNP6MBSACDEXHE73QZM5T3ANCNFSM4J4MJY5A>
> .
>

If we want to read this:

```
   <CalibrationUnits>
     <Name>V</Name>
     <Description></Description>       // -or it could be: <Description/>
 -or: <Description> </Description> with same result
    </CalibrationUnits>
    <Sensor>
     <Description/>
    </Sensor>
```

and have it write this back out as:

```
        <CalibrationUnits>
          <Name>V</Name>
          <Description></Description>
        </CalibrationUnits>
        <Sensor>
          <Description></Description>
        </Sensor>
```

e.g., the only way the output stationxml won't have the Sensor:Description
is if it is completely missing from the input.

The following changes to io/stationxml/core.py will do it:

```
import re
def _strip_white(string):
    return(re.sub(r'\s+', '', string))

def _tag2obj(element, tag, convert):
    # we use future.builtins.str and are sure we have unicode here
    try:
        text = element.find(tag).text
        if text is None or len(_strip_white(text)) == 0:
            return '''
     // Returning an empty string
        else:
            return convert(element.find(tag).text)
    except Exception:
        None

1129     if channel.calibration_units:
1130         cu = etree.SubElement(channel_elem, "CalibrationUnits")
1131         etree.SubElement(cu, "Name").text = \
1132             str(channel.calibration_units)
1133         #if channel.calibration_units_description:
1134         if channel.calibration_units_description is not None:
// This is the only change, so that it only omits description if it wasn't
in the input
1135             etree.SubElement(cu, "Description").text = \
1136                 str(channel.calibration_units_description)
```

Let me know what you think - I can fork the repo and put these changes into
a new branch (tagged with the issue ?)

Cheers,
-Mike


@mikehagerty  go ahead..

The only thing is I would tend to think that an empty string shouldn't be the same as a string with two spaces. I think we should play it safe and initialize a `Description` with a value of two spaces in that case.

```
     <Description></Description>       // -or it could be: <Description/>
 -or: <Description> </Description> with same result
```



Resolving patch:
diff --git a/obspy/io/seiscomp/inventory.py b/obspy/io/seiscomp/inventory.py
--- a/obspy/io/seiscomp/inventory.py
+++ b/obspy/io/seiscomp/inventory.py
@@ -536,7 +536,7 @@ def _read_instrument_sensitivity(sen_element, cha_element, _ns):
     frequency = _tag2obj(cha_element, _ns("gainFrequency"), float)
 
     input_units_name = _tag2obj(sen_element, _ns("unit"), str)
-    output_units_name = str(None)
+    output_units_name = ''
 
     sensitivity = obspy.core.inventory.response.InstrumentSensitivity(
         value=gain, frequency=frequency,
diff --git a/obspy/io/stationxml/core.py b/obspy/io/stationxml/core.py
--- a/obspy/io/stationxml/core.py
+++ b/obspy/io/stationxml/core.py
@@ -1445,18 +1445,37 @@ def _write_extra(parent, obj):
             _write_element(parent, element, tagname)
 
 
-def _tag2obj(element, tag, convert):
-    # we use future.builtins.str and are sure we have unicode here
+def _convert(text, convert):
+    if convert is str:
+        return _convert_str(text)
     try:
-        return convert(element.find(tag).text)
+        return convert(text)
     except Exception:
         None
 
 
+def _convert_str(text):
+    # we use future.builtins.str and are sure we have unicode here
+    # lxml gives ``None`` for tags with empty text, best way to handle this is
+    # probably to give back an empty string
+    if not text:
+        # Returning an empty string
+        return ''
+    return text
+
+
+def _tag2obj(element, tag, convert):
+    try:
+        text = element.find(tag).text
+    except Exception:
+        return None
+    return _convert(text, convert)
+
+
 def _tags2obj(element, tag, convert):
     values = []
     for elem in element.findall(tag):
-        values.append(convert(elem.text))
+        values.append(_convert(elem.text, convert))
     return values
 
 


Test patch: