Repository: pydicom/pydicom

Issue:
Inconsistencies in value testing for PersonName3
```python
from pydicom.dataset import Dataset

ds = Dataset()
ds.PatientName = None  # or ''
if ds.PatientName:
    print('Has a value')
else:
    print('Has no value')

if None:  # or ''
    print('Evaluates as True')
else:
    print('Evaluates as False')
```
Prints `Has a value` then `Evaluates as False`. Should print `Has no value` instead (encoded dataset will have a zero-length element).

Current master, python 3.6.


Discussion:
Should that `if None` read `if ds.PatientName is None`?

I can see the inconsistency if assigning `None` - which results in an empty value instead of `None`. Assigning an empty string works as expected (has an empty string afterwards).
For other string tags `None` or `''` can be assigned, though I now wonder if this is the correct behavior, as there is no difference between an empty value and no value in DICOM for string VRs. IMHO, the correct behavior should be to assign no value, e.g. the following should be equivalent:
```python
ds.PatientName = ''
ds.PatientName = None
ds.PatientName = []
```
Currently, the first 2 result in VM=1, the last in VM=0 (which is correct).
@darcymason, @scaramallion - what do you think?
The `if None` was just to demonstrate how python behaves and hence how I naively expect an empty pydicom element value to behave.

I just want a simple consistent "does this element have a value" test, which means an element with `None` and `''` should both fail the test. Essentially if the element will end up encoded with zero-length it should fail. It works as I expect for simple text element values, but `PersonName3` and maybe some of the other VR classes might need `__bool__()` or `__len__()`?

```python
# I'd like this to work cleanly
for elem in ds:
   if elem.value:
       print(elem.value)
   else:
       print('(no value available)')
```

I've been playing around with printing out pretty datasets for pynetdicom and have run into a some problems with consistency in how element values are evaluated and printed.

I think you're right, they should be VM = 0. I think the last case gives you a `MultiValue` rather than a `PersonName3` which is probably why.
```python
#!/usr/bin/env python

from pydicom.dataset import Dataset
from pydicom.datadict import tag_for_keyword

VRs = {
    'AE' : 'Receiver',
    'AS' : 'PatientAge',
    'AT' : 'OffendingElement',
    'CS' : 'QualityControlSubject',
    'DA' : 'PatientBirthDate',
    'DS' : 'PatientWeight',
    'DT' : 'AcquisitionDateTime',
    'IS' : 'BeamNumber',
    'LO' : 'DataSetSubtype',
    'LT' : 'ExtendedCodeMeaning',
    'PN' : 'PatientName',
    'SH' : 'CodeValue',
    'ST' : 'InstitutionAddress',
    'TM' : 'StudyTime',
    'UC' : 'LongCodeValue',
    'UI' : 'SOPClassUID',
    'UR' : 'CodingSchemeURL',
    'UT' : 'StrainAdditionalInformation',
    'SL' : 'RationalNumeratorValue',
    'SS' : 'SelectorSSValue',
    'UL' : 'SimpleFrameList',
    'US' : 'SourceAcquisitionBeamNumber',
    'FD' : 'RealWorldValueLUTData',
    'FL' : 'VectorAccuracy',
    'OB' : 'FillPattern',
    'OD' : 'DoubleFloatPixelData',
    'OF' : 'UValueData',
    'OL' : 'TrackPointIndexList',
    'OW' : 'TrianglePointIndexList',
    #'OV' : '',
    'UN' : 'SelectorUNValue',
}

ds = Dataset()
for vr, kw in VRs.items():
    try:
        setattr(ds, kw, None)
    except:
        print('Failed assigning None', vr)
        continue

    elem = ds[tag_for_keyword(kw)]

    try:
        assert bool(elem.value) is False
    except:
        print('Failed empty value test', vr)

    try:
        assert elem.VM == 0
    except Exception as exc:
        print('Failed VM == 0', vr, elem.VM)
```
UI fails assignment using `None`, they all fail the VM == 0 test, PN fails the value test in Python 3.6, passes in 2.7.
PN in Python 2 is derived from `unicode`, so this is to be expected. So you agree that VM shall be 0 after assigning `None` or `''`?
> So you agree that VM shall be 0 after assigning None or ''?

As far as I can see the standard doesn't explicitly say so, but I think that's reasonable. DCMTK's `dcmdump` uses that convention.

What do you think @darcymason?
> I think you're right, they should be VM = 0. I think the last case gives you a MultiValue rather than a PersonName3 which is probably why.

Sorry, I overlooked that post. Yes, and that may be the easiest way to change this - assign an empty `MultiValue` (aka an empty list) in these cases instead of `None` or `''`.
There is also the question of how to represent empty values. Currently we get:
```
>>> ds.CodeValue = None
>>> ds.CodeValue
>>> print(ds.CodeValue)
None
>>> ds['CodeValue'].VM
1
>>> ds.CodeValue = []
>>> ds.CodeValue
['']
>>> ds['CodeValue'].VM
0
```
which is quite inconsistent. `None` for an empty value may make sense, or an empty list (other than a list with an empty string), not sure about this. Any ideas?
I think as long as `bool(value)` would normally evaluate as `False` then it should be considered empty.
```python
value = '' # or None or [] or b'' or whatever
if not value:
    print('Empty')
```
For `MultiValue` it should probably be an OR on `[bool(val) for val in MultiValue]`
> > So you agree that VM shall be 0 after assigning None or ''?
> 
> As far as I can see the standard doesn't explicitly say so, but I think that's reasonable. DCMTK's `dcmdump` uses that convention.
> 
> What do you think @darcymason?

Hmmm... just read quickly through, so apologies if I've repeated (or missed) something already said.  Haven't looked at any code yet.

One thought is that DICOM type 2 data elements allow blank values, which to me is a real valid value, different than a None.

Second thought is we've normally handled a non-existent value by just not having the attribute set, i.e. just don't set the attribute, or if it already exists, delete it.  You check for existence of values by `if keyword in ds`.

Third thought (the last in this comment I think) is that when we drop support for earlier python versions and can start typing pydicom, do we cause trouble by encouraging use of None?  We have not forbidden setting None, or even incorrect types in many cases, but this is in line with python's duck typing philosophy.  It may be that at some point more control over this could be automatically introduced via typing. Yes, we could Union everything with None, but I would foresee a whole lot of `if` statements to handle those cases.


I frequently use `None` for type 2 elements that are required to be present but may or may not have a value. It just seems to me that this is the 'obvious' thing to do when I want an element with no value. Does it make sense that you need to set an integer VR to `''` if you want an empty value?

`if keyword in ds` tests for the presence of the element in the dataset, not that the element has a value. 
Well, thinking this through some more, I may be coming around to your way of thinking...I suppose blank type 2 can be considered a zero-length (no value) data element.  Using a null (python None) value for those does have a certain logic to it, similar to Null in databases and so on.  And in that case, VM should be zero, although VM is a weak concept to me anyway, since it exists in dictionaries but not in actual DICOM files, so I'm not too invested in that one way or the other.
The other thing is that some elements don't encode properly with a value set to `None`, while others work as I expect.

```python
>>> from pydicom.dataset import Dataset
>>> from pynetdicom.dsutils import encode  # wrapper for filewriter.write_dataset
>>> ds = Dataset()
>>> ds.Receiver = None  # VR of AE
>>> type(ds.Receiver)
NoneType
>>> encode(ds, True, True)  # Works OK
b'\x00\x00\x00\x03\x00\x00\x00\x00'
>>> ds = Dataset()
>>> ds.BitsStored = None  # VR of US
>>> type(ds.BitsStored)
NoneType
>>> encode(ds, True, True)  # Fails to encode
>>>
```
So I think this has elevated to a larger conversation on "`None` support" in pydicom, make it consistent across all types, or raise exceptions if there are any types or situations where it should not be allowed.

I'll sleep on it for now, pick it up again tomorrow...

My current instinct is to:
- handle assigning `None`, `''` or `[]` to an element with a text VR as assigning an empty string (as they are all equivalent, and an empty string is most convienient for the user, and probably expected), return `False` as truth value, and have a VM of 0 (this is as implemented in the PR) 
- handle assigning `None` or `[]` to an element with a binary VR as assigning `None` (I think this better meets the expections for an element without a value), return `False` as truth value, and have a VM of 0 (in the PR I use a `MultiValue` object, which I would change)
- in the case of IS and DS (these are kind of in-between), we have to make sure they handle the empty string case by returning `None` instead of 0 if getting the numerical value (haven't checked the current behavior) 

Using `None` for text values, apart from not being upwards compatible, is a thing I have been burned with before. I used to write DICOM elements into a database and had used database `null` to represent elements without values. This turned out to be a big mistake, as there is actually no use case where you have to differentiate no value from an empty value for text values (and indeed there is no such distinction in DICOM itself for text VRs), and led to a lot of unnecessary checking for null values in the user code (and exceptions if they have been missing).
I've started a separate issue for the empty values discussion #896

Patch:
diff --git a/pydicom/dataelem.py b/pydicom/dataelem.py
--- a/pydicom/dataelem.py
+++ b/pydicom/dataelem.py
@@ -378,7 +378,7 @@ def _convert(self, val):
         elif self.VR == 'TM' and config.datetime_conversion:
             return pydicom.valuerep.TM(val)
         elif self.VR == "UI":
-            return UID(val)
+            return UID(val if val else '')
         elif not in_py2 and self.VR == "PN":
             return PersonName(val)
         # Later may need this for PersonName as for UI,
diff --git a/pydicom/multival.py b/pydicom/multival.py
--- a/pydicom/multival.py
+++ b/pydicom/multival.py
@@ -2,6 +2,7 @@
 """Code for multi-value data elements values,
 or any list of items that must all be the same type.
 """
+from pydicom import compat
 
 try:
     from collections.abc import MutableSequence
@@ -30,7 +31,7 @@ def __init__(self, type_constructor, iterable):
         :param type_constructor: a constructor for the required
                            type for all list items. Could be the
                            class, or a factory function. For DICOM
-                           mult-value data elements, this will be the
+                           multi-value data elements, this will be the
                            class or type corresponding to the VR.
         :param iterable: an iterable (e.g. list, tuple) of items
                         to initialize the MultiValue list
@@ -59,8 +60,11 @@ def __setitem__(self, i, val):
             self._list.__setitem__(i, self.type_constructor(val))
 
     def __str__(self):
-        lines = [str(x) for x in self]
-        return "['" + "', '".join(lines) + "']"
+        if not self:
+            return ''
+        lines = ["'{}'".format(x) if isinstance(x, compat.char_types)
+                 else str(x) for x in self]
+        return "[" + ", ".join(lines) + "]"
 
     __repr__ = __str__
 
diff --git a/pydicom/valuerep.py b/pydicom/valuerep.py
--- a/pydicom/valuerep.py
+++ b/pydicom/valuerep.py
@@ -603,8 +603,6 @@ def _encode_personname(components, encodings):
 
 class PersonName3(object):
     def __init__(self, val, encodings=None, original_string=None):
-        # handle None `val` as empty string
-        val = val or ''
         if isinstance(val, PersonName3):
             encodings = val.encodings
             self.original_string = val.original_string
@@ -614,6 +612,9 @@ def __init__(self, val, encodings=None, original_string=None):
             self.original_string = val
             self._components = None
         else:
+            # handle None `val` as empty string
+            val = val or ''
+
             # this is the decoded string - save the original string if
             # available for easier writing back
             self.original_string = original_string
@@ -774,6 +775,12 @@ def formatted(self, format_str):
         self._create_dict()
         return format_str % self._dict
 
+    def __bool__(self):
+        if self.original_string is None:
+            return (self._components is not None and
+                    (len(self._components) > 1 or bool(self._components[0])))
+        return bool(self.original_string)
+
 
 class PersonNameBase(object):
     """Base class for Person Name classes"""


Test patch:
diff --git a/pydicom/tests/test_dataelem.py b/pydicom/tests/test_dataelem.py
--- a/pydicom/tests/test_dataelem.py
+++ b/pydicom/tests/test_dataelem.py
@@ -15,6 +15,7 @@
     DataElement_from_raw,
 )
 from pydicom.dataset import Dataset
+from pydicom.multival import MultiValue
 from pydicom.tag import Tag
 from pydicom.uid import UID
 from pydicom.valuerep import DSfloat
@@ -368,6 +369,92 @@ def test_private_repeater_tag(self):
         assert '[Overlay ID]' == private_data_elem.name
         assert 'UN' == private_data_elem.VR
 
+    def test_empty_text_values(self):
+        """Test that assigning an empty value behaves as expected."""
+        def check_empty_text_element(value):
+            setattr(ds, tag_name, value)
+            elem = ds[tag_name]
+            assert bool(elem.value) is False
+
+        text_vrs = {
+            'AE': 'Receiver',
+            'AS': 'PatientAge',
+            'AT': 'OffendingElement',
+            'CS': 'QualityControlSubject',
+            'DA': 'PatientBirthDate',
+            'DS': 'PatientWeight',
+            'DT': 'AcquisitionDateTime',
+            'IS': 'BeamNumber',
+            'LO': 'DataSetSubtype',
+            'LT': 'ExtendedCodeMeaning',
+            'PN': 'PatientName',
+            'SH': 'CodeValue',
+            'ST': 'InstitutionAddress',
+            'TM': 'StudyTime',
+            'UC': 'LongCodeValue',
+            'UI': 'SOPClassUID',
+            'UR': 'CodingSchemeURL',
+            'UT': 'StrainAdditionalInformation',
+        }
+        ds = Dataset()
+        # set value to new element
+        for tag_name in text_vrs.values():
+            check_empty_text_element(None)
+            del ds[tag_name]
+            check_empty_text_element(b'')
+            del ds[tag_name]
+            check_empty_text_element(u'')
+            del ds[tag_name]
+            check_empty_text_element([])
+            del ds[tag_name]
+
+        # set value to existing element
+        for tag_name in text_vrs.values():
+            check_empty_text_element(None)
+            check_empty_text_element(b'')
+            check_empty_text_element(u'')
+            check_empty_text_element([])
+            check_empty_text_element(None)
+
+    def test_empty_binary_values(self):
+        """Test that assigning an empty value behaves as expected for
+        non-text VRs."""
+        def check_empty_binary_element(value):
+            setattr(ds, tag_name, value)
+            elem = ds[tag_name]
+            assert bool(elem.value) is False
+
+        non_text_vrs = {
+            'SL': 'RationalNumeratorValue',
+            'SS': 'SelectorSSValue',
+            'UL': 'SimpleFrameList',
+            'US': 'SourceAcquisitionBeamNumber',
+            'FD': 'RealWorldValueLUTData',
+            'FL': 'VectorAccuracy',
+            'OB': 'FillPattern',
+            'OD': 'DoubleFloatPixelData',
+            'OF': 'UValueData',
+            'OL': 'TrackPointIndexList',
+            'OW': 'TrianglePointIndexList',
+            'UN': 'SelectorUNValue',
+        }
+        ds = Dataset()
+        # set value to new element
+        for tag_name in non_text_vrs.values():
+            check_empty_binary_element(None)
+            del ds[tag_name]
+            check_empty_binary_element([])
+            del ds[tag_name]
+            check_empty_binary_element(MultiValue(int, []))
+            del ds[tag_name]
+
+        # set value to existing element
+        for tag_name in non_text_vrs.values():
+            check_empty_binary_element(None)
+            check_empty_binary_element([])
+            check_empty_binary_element(MultiValue(int, []))
+            check_empty_binary_element(None)
+
 
 class RawDataElementTests(unittest.TestCase):
     def testKeyError(self):
diff --git a/pydicom/tests/test_multival.py b/pydicom/tests/test_multival.py
--- a/pydicom/tests/test_multival.py
+++ b/pydicom/tests/test_multival.py
@@ -1,43 +1,46 @@
 # Copyright 2008-2018 pydicom authors. See LICENSE file for details.
 """Test suite for MultiValue class"""
 
-import unittest
+import pytest
+
 from pydicom.multival import MultiValue
 from pydicom.valuerep import DS, DSfloat, DSdecimal, IS
-from pydicom import config
+from pydicom import config, compat
 from copy import deepcopy
 
 import sys
+
 python_version = sys.version_info
 
 
-class MultiValuetests(unittest.TestCase):
+class TestMultiValue(object):
     def testMultiDS(self):
         """MultiValue: Multi-valued data elements can be created........"""
         multival = MultiValue(DS, ['11.1', '22.2', '33.3'])
         for val in multival:
-            self.assertTrue(isinstance(val, (DSfloat, DSdecimal)),
-                            "Multi-value DS item not converted to DS")
+            assert isinstance(val, (DSfloat, DSdecimal))
 
     def testEmptyElements(self):
         """MultiValue: Empty number string elements are not converted..."""
         multival = MultiValue(DSfloat, ['1.0', ''])
-        self.assertEqual(1.0, multival[0])
-        self.assertEqual('', multival[1])
+        assert 1.0 == multival[0]
+        assert '' == multival[1]
         multival = MultiValue(IS, ['1', ''])
-        self.assertEqual(1, multival[0])
-        self.assertEqual('', multival[1])
+        assert 1 == multival[0]
+        assert '' == multival[1]
         multival = MultiValue(DSdecimal, ['1', ''])
-        self.assertEqual(1, multival[0])
-        self.assertEqual('', multival[1])
+        assert 1 == multival[0]
+        assert '' == multival[1]
+        multival = MultiValue(IS, [])
+        assert not multival
+        assert 0 == len(multival)
 
     def testLimits(self):
         """MultiValue: Raise error if any item outside DICOM limits...."""
         original_flag = config.enforce_valid_values
         config.enforce_valid_values = True
-        self.assertRaises(OverflowError,
-                          MultiValue,
-                          IS, [1, -2 ** 31 - 1])
+        with pytest.raises(OverflowError):
+            MultiValue(IS, [1, -2 ** 31 - 1])
         # Overflow error not raised for IS out of DICOM valid range
         config.enforce_valid_values = original_flag
 
@@ -45,44 +48,39 @@ def testAppend(self):
         """MultiValue: Append of item converts it to required type..."""
         multival = MultiValue(IS, [1, 5, 10])
         multival.append('5')
-        self.assertTrue(isinstance(multival[-1], IS))
-        self.assertEqual(multival[-1], 5,
-                         "Item set by append is not correct value")
+        assert isinstance(multival[-1], IS)
+        assert multival[-1] == 5
 
     def testSetIndex(self):
         """MultiValue: Setting list item converts it to required type"""
         multival = MultiValue(IS, [1, 5, 10])
         multival[1] = '7'
-        self.assertTrue(isinstance(multival[1], IS))
-        self.assertEqual(multival[1], 7,
-                         "Item set by index is not correct value")
+        assert isinstance(multival[1], IS)
+        assert multival[1] == 7
 
     def testDeleteIndex(self):
         """MultiValue: Deleting item at index behaves as expected..."""
         multival = MultiValue(IS, [1, 5, 10])
         del multival[1]
-        self.assertEqual(2, len(multival))
-        self.assertEqual(multival[0], 1)
-        self.assertEqual(multival[1], 10)
+        assert 2 == len(multival)
+        assert multival[0] == 1
+        assert multival[1] == 10
 
     def testExtend(self):
         """MultiValue: Extending a list converts all to required type"""
         multival = MultiValue(IS, [1, 5, 10])
         multival.extend(['7', 42])
-        self.assertTrue(isinstance(multival[-2], IS))
-        self.assertTrue(isinstance(multival[-1], IS))
-        self.assertEqual(multival[-2], 7,
-                         "Item set by extend not correct value")
+        assert isinstance(multival[-2], IS)
+        assert isinstance(multival[-1], IS)
+        assert multival[-2], 7
 
     def testSlice(self):
         """MultiValue: Setting slice converts items to required type."""
         multival = MultiValue(IS, range(7))
         multival[2:7:2] = [4, 16, 36]
         for val in multival:
-            self.assertTrue(isinstance(val, IS),
-                            "Slice IS value not correct type")
-        self.assertEqual(multival[4], 16,
-                         "Set by slice failed for item 4 of list")
+            assert isinstance(val, IS)
+            assert multival[4] == 16
 
     def testIssue236DeepCopy(self):
         """MultiValue: deepcopy of MultiValue does not generate an error"""
@@ -97,38 +95,45 @@ def testSorting(self):
         """MultiValue: allow inline sort."""
         multival = MultiValue(DS, [12, 33, 5, 7, 1])
         multival.sort()
-        self.assertEqual([1, 5, 7, 12, 33], multival)
+        assert [1, 5, 7, 12, 33] == multival
         multival.sort(reverse=True)
-        self.assertEqual([33, 12, 7, 5, 1], multival)
+        assert [33, 12, 7, 5, 1] == multival
         multival.sort(key=str)
-        self.assertEqual([1, 12, 33, 5, 7], multival)
+        assert [1, 12, 33, 5, 7] == multival
 
     def test_equal(self):
         """MultiValue: test equality operator"""
         multival = MultiValue(DS, [12, 33, 5, 7, 1])
         multival2 = MultiValue(DS, [12, 33, 5, 7, 1])
         multival3 = MultiValue(DS, [33, 12, 5, 7, 1])
-        self.assertTrue(multival == multival2)
-        self.assertFalse(multival == multival3)
+        assert multival == multival2
+        assert not (multival == multival3)
         multival = MultiValue(str, ['a', 'b', 'c'])
         multival2 = MultiValue(str, ['a', 'b', 'c'])
         multival3 = MultiValue(str, ['b', 'c', 'a'])
-        self.assertTrue(multival == multival2)
-        self.assertFalse(multival == multival3)
+        assert multival == multival2
+        assert not (multival == multival3)
 
     def test_not_equal(self):
         """MultiValue: test equality operator"""
         multival = MultiValue(DS, [12, 33, 5, 7, 1])
         multival2 = MultiValue(DS, [12, 33, 5, 7, 1])
         multival3 = MultiValue(DS, [33, 12, 5, 7, 1])
-        self.assertFalse(multival != multival2)
-        self.assertTrue(multival != multival3)
+        assert not multival != multival2
+        assert multival != multival3
         multival = MultiValue(str, ['a', 'b', 'c'])
         multival2 = MultiValue(str, ['a', 'b', 'c'])
         multival3 = MultiValue(str, ['b', 'c', 'a'])
-        self.assertFalse(multival != multival2)
-        self.assertTrue(multival != multival3)
-
-
-if __name__ == "__main__":
-    unittest.main()
+        assert not (multival != multival2)
+        assert multival != multival3
+
+    def test_str_rep(self):
+        """MultiValue: test print output"""
+        multival = MultiValue(IS, [])
+        assert '' == str(multival)
+        multival = MultiValue(compat.text_type, [1, 2, 3])
+        assert "['1', '2', '3']" == str(multival)
+        multival = MultiValue(int, [1, 2, 3])
+        assert '[1, 2, 3]' == str(multival)
+        multival = MultiValue(float, [1.1, 2.2, 3.3])
+        assert '[1.1, 2.2, 3.3]' == str(multival)