Repository: pydicom/pydicom

Issue:
Strict adherence to VR during parsing is detrimental due to commonplace vendor interpretations
**Describe the bug**
DICOM Files from GE modalities, which when parsed, raise a TypeError caused by "violating" the VR imposed by the DICOM standard; however, real world modalities have and continue to generate such files for good cause.

For example the following is raised

`TypeError('Could not convert value to integer without loss')`

by a real world DICOM file which has a value

`(0018,1152) IS [14.5]                                   #   4, 1 Exposure`

where IS is a Value Representation defined as

> IS - Integer String

> A string of characters representing an Integer in base-10 (decimal), shall contain only the characters 0 - 9, with an optional leading "+" or "-". It may be padded with leading and/or trailing spaces. Embedded spaces are not allowed.

> The integer, n, represented shall be in the range: -231<= n <= (231-1).

[See DICOM Part 5 Section 6.2](https://dicom.nema.org/dicom/2013/output/chtml/part05/sect_6.2.html)

which means `14.5` is an invalid value due to the fractional portion .5 which definitely would lead to a loss in precision if converted to a pure integer value (of 14). 

After discussion with a senior engineer for the vendor, the following dialogue was obtained which quotes an article by David Clune, a well-respected, long-time member of the DICOM committee and community:

> The tag pair in question is meant to contain the mAs value used for the exposure, which is not constrained to integer values, but for some reason the DICOM standard defines it as such.

> An interesting article from someone responsible for maintaining the DICOM documentation explains the conundrum quite well:  

http://dclunie.blogspot.com/2008/11/dicom-exposure-attribute-fiasco.html

> Of note are two excerpts from that article:

> "The original ACR-NEMA standard specified ASCII numeric data elements for Exposure, Exposure Time and X-Ray Tube Current that could be decimal values; for no apparent reason DICOM 3.0 in 1993 constrained these to be integers, which for some modalities and subjects are too small to be sufficiently precise"

> and

> "The authors of DICOM, in attempting to maintain some semblance of backward compatibility with ACR-NEMA and at the same time apply more precise constraints, re-defined all ACR-NEMA data elements of VR AN as either IS or DS, the former being the AN integer numbers (with new size constraints), and the latter being the AN fixed point and floating point numbers. In the process of categorizing the old data elements into either IS or DS, not only were the obvious integers (like counts of images and other things) made into integers, but it appears that also any "real world" attribute that in somebody's expert opinion did not need greater precision than a whole integer, was so constrained as well."

> I have inspected a few random DICOM files generated by various modalities and the value is stored accurately, even though it is a violation of the explicit value representation. Additionally, I have worked with (and support) various PACS platforms, and this is the first time this has been raised as an issue. So technically, you are correct that encoding that value as decimal violates the explicit VR, but it appears to be common practice to do so. 

**Expected behavior**
To deal with the reality of history with respect to the current standard, my opinion, as a long-standing DICOM PACS implementer at Medstrat, is that there is nothing to gain and everything to lose by raising a `TypeError` here. For cases where an integer VR, such as `IS`, could be read as a floating point number instead, then it should be allowed to be so, for at least a limited whitelist of tags.

Arguments against which come to mind are of the ilk that do not heed "Although practicality beats purity" as can be read if you 

[`>>> import this`](https://peps.python.org/pep-0020/)

> Special cases aren't special enough to break the rules.
> Although practicality beats purity.

**Steps To Reproduce**

`(0018,1152) IS [14.5]                                   #   4, 1 Exposure`

Set any DICOM file to have the above for `Exposure` and then do this:

```
>>> from pydicom import config
>>> pydicom.__version__
'2.3.0'
>>> config.settings.reading_validation_mode = config.IGNORE
>>> ds = pydicom.dcmread('1.2.840.113619.2.107.20220429121335.1.1.dcm')
>>> ds
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/pydicom/dataset.py", line 2306, in __str__
    return self._pretty_str()
  File "/usr/local/lib/python3.7/site-packages/pydicom/dataset.py", line 2020, in _pretty_str
    for elem in self:
  File "/usr/local/lib/python3.7/site-packages/pydicom/dataset.py", line 1240, in __iter__
    yield self[tag]
  File "/usr/local/lib/python3.7/site-packages/pydicom/dataset.py", line 939, in __getitem__
    self[tag] = DataElement_from_raw(elem, character_set, self)
  File "/usr/local/lib/python3.7/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw
    value = convert_value(vr, raw, encoding)
  File "/usr/local/lib/python3.7/site-packages/pydicom/values.py", line 771, in convert_value
    return converter(byte_string, is_little_endian, num_format)
  File "/usr/local/lib/python3.7/site-packages/pydicom/values.py", line 348, in convert_IS_string
    return MultiString(num_string, valtype=pydicom.valuerep.IS)
  File "/usr/local/lib/python3.7/site-packages/pydicom/valuerep.py", line 1213, in MultiString
    return valtype(splitup[0])
  File "/usr/local/lib/python3.7/site-packages/pydicom/valuerep.py", line 1131, in __new__
    raise TypeError("Could not convert value to integer without loss")
TypeError: Could not convert value to integer without loss
```

**Your environment**

```bash
module       | version
------       | -------
platform     | Darwin-21.5.0-x86_64-i386-64bit
Python       | 3.7.5 (v3.7.5:5c02a39a0b, Oct 14 2019, 18:49:57)  [Clang 6.0 (clang-600.0.57)]
pydicom      | 2.2.2
gdcm         | _module not found_
jpeg_ls      | _module not found_
numpy        | _module not found_
PIL          | _module not found_
pylibjpeg    | _module not found_
openjpeg     | _module not found_
libjpeg      | _module not found_
```



Discussion:
Thank you for that thorough explanation (and of course we all know and respect David Clunie 😄 )!
I understand the specific problem with `Exposure Time` (for a similar reason, there exists the tag `Exposure in μAs` additionally to `Exposure`, but no such thing exists for `Exposure Time`), and I am aware that this is not the only case where float values are written into `IS`. 

I think a similar issue came up before, and my preference would be to make this behavior dependent on the validation mode. The default validation mode for reading is to issue a warning, while for writing new data it is to raise an exception. This would not resolve the problem, though. The real problem is in the implementation of the tag class: the class representing `IS` values is derived from `int`. In my opinion, it would make sense to change that, though I'm not sure if we can do this consistently in a backwards-compatible way.

@darcymason - I'm quite sure that we have discussed this before, I will try to find the respective issue. Anyway, what are your current thoughts here? 

CC @dclunie

Ah ok, @gsmethells has already [commented](https://github.com/pydicom/pydicom/issues/1643#issuecomment-1180789608) in the respective issue #1643.
> I'm quite sure that we have discussed this before

Yes, it has come up several times.

A new thought has come to me (maybe not thought through properly, or maybe someone has mentioned this before and I'm just thinking it is new): maybe we could create an ISFloat class, operating similarly to the DSFloat idea.  Then, just have an allow_IS_float config flag, perhaps even true by default (return an ISFloat only if it is not an exact int).  Python allows mixed math anyway, so I don't see any real problem in returning a non-int if further math is done.   Only problem might be code that did an `isinstance` check against `IS` but that should be very rare.
> maybe we could create an ISFloat class, operating similarly to the DSFloat idea

My preference would be to change the `IS` class to support both `int` and `float`. We cannot always use float, as that could decrease the precision of integers, but in the case that float numbers are written in the tag, I would prefer to return a float. Not sure what problems this would bring, though... 
Additionally, I think we can still couple the behavior to the validation mode, but I'm not completely sure here.
> My preference would be to change the `IS` class to support both `int` and `float`.

Is that actually possible to do, without recreating all the class methods for `int` (or `float`) for math operations?
> Is that actually possible to do, without recreating all the class methods for int (or float) for math operations?

I guess not - that would be the downside of that approach. Also an `isinstance(int)` would fail, of course. It is probably better to use `ISFloat` as you proposed and dynamically decide which class to use.
> It is probably better to use ISFloat as you proposed and dynamically decide which class to use.

Actually that is what you have proposed - sorry, I misread that, I understood that you wanted to configure which class to use. Yes, I like your proposal!
> Is that actually possible to do, without recreating all the class methods for int (or float) for math operations?

Actually, I feel like we did return a different class from `__new__` somewhere, or at least talked about it.  It turns out that it is possible to [return a different class from `__new__`](https://stackoverflow.com/questions/20221858/python-new-method-returning-something-other-than-class-instance).  I'm not sure it is advisable, though, that is really not being explicit to the user.
> Actually, I feel like we did return a different class from `__new__` somewhere

Yes, we actually use this to return either a string or an int/float from `IS`/`DS`. This is also a common pattern in Python (they use it for example in `pathlib`), so I think it would not be unexpected.
Thank you for taking the time to discuss ideas and consider this. I understand wanting to adhere tightly to the standard (we do so for edge cases the majority of the time ourselves for our ortho PACS). I also understand the desire to listen to the "import this" zen of "Simple is better than complex" and, yes, I understand all too well how a large system can grow complicated.

My usual pause comes when my design desires run up against DICOM files in the field where there is a common violation of the standard. We maintain petabytes of DICOM images and this issue is common. I would be happy if reading/writing `dataset`s from/to files via pydicom continued to support maintaining existing values, regardless of whether those values violate the standard, (it does now, but is that an intentional design decision? A `dataset` only raises a `TypeError` when the value is directly attempted to be read out, whether via `__iter__` or otherwise) and also provided a (new/existing?) preference for reading `int` VR tag values as `float`.

My two cents.
> reading/writing `dataset`s from/to files via pydicom continued to support maintaining existing values, regardless of whether those values violate the standard

Any other position would become a deal breaker for our use of the library if it were ever otherwise, simply due to the realities of supporting customers. Especially when it comes to the need to update tags (e.g., a misspelled PatientName) while keeping other unrelated tags the same (i.e., no side-effects).
Setting 2.4 milestone to at least consider this for the release.

Patch:
diff --git a/pydicom/config.py b/pydicom/config.py
--- a/pydicom/config.py
+++ b/pydicom/config.py
@@ -268,6 +268,18 @@ def disable_value_validation() -> Generator:
         settings._writing_validation_mode = writing_mode
 
 
+@contextmanager
+def strict_reading() -> Generator:
+    """Context manager to temporarily enably strict value validation
+    for reading."""
+    original_reading_mode = settings._reading_validation_mode
+    try:
+        settings.reading_validation_mode = RAISE
+        yield
+    finally:
+        settings._reading_validation_mode = original_reading_mode
+
+
 convert_wrong_length_to_UN = False
 """Convert a field VR to "UN" and return bytes if bytes length is invalid.
 Default ``False``.
diff --git a/pydicom/dataset.py b/pydicom/dataset.py
--- a/pydicom/dataset.py
+++ b/pydicom/dataset.py
@@ -16,6 +16,7 @@
 """
 import copy
 from bisect import bisect_left
+from contextlib import nullcontext
 import io
 from importlib.util import find_spec as have_package
 import inspect  # for __dir__
@@ -2490,18 +2491,23 @@ def to_json_dict(
             :class:`Dataset` representation based on the DICOM JSON Model.
         """
         json_dataset = {}
-        for key in self.keys():
-            json_key = '{:08X}'.format(key)
-            try:
-                data_element = self[key]
-                json_dataset[json_key] = data_element.to_json_dict(
-                    bulk_data_element_handler=bulk_data_element_handler,
-                    bulk_data_threshold=bulk_data_threshold
-                )
-            except Exception as exc:
-                logger.error(f"Error while processing tag {json_key}")
-                if not suppress_invalid_tags:
-                    raise exc
+        context = (
+            config.strict_reading() if suppress_invalid_tags
+            else nullcontext()
+        )
+        with context:
+            for key in self.keys():
+                json_key = '{:08X}'.format(key)
+                try:
+                    data_element = self[key]
+                    json_dataset[json_key] = data_element.to_json_dict(
+                        bulk_data_element_handler=bulk_data_element_handler,
+                        bulk_data_threshold=bulk_data_threshold
+                    )
+                except Exception as exc:
+                    logger.error(f"Error while processing tag {json_key}")
+                    if not suppress_invalid_tags:
+                        raise exc
 
         return json_dataset
 
diff --git a/pydicom/valuerep.py b/pydicom/valuerep.py
--- a/pydicom/valuerep.py
+++ b/pydicom/valuerep.py
@@ -1248,6 +1248,48 @@ def DS(
     return DSfloat(val, auto_format, validation_mode)
 
 
+class ISfloat(float):
+    """Store value for an element with VR **IS** as :class:`float`.
+
+    Stores original integer string for exact rewriting of the string
+    originally read or stored.
+
+    Note: By the DICOM standard, IS can only be an :class:`int`,
+    however, it is not uncommon to see float IS values.  This class
+    is used if the config settings allow non-strict reading.
+
+    Generally, use :class:`~pydicom.valuerep.IS` to create IS values,
+    this is returned instead if the value cannot be represented as an
+    :class:`int`.  See :class:`~pydicom.valuerep.IS` for details of the
+    parameters and return values.
+    """
+    def __new__(  # type: ignore[misc]
+            cls: Type["ISfloat"], val: Union[str, float, Decimal],
+            validation_mode: int = None
+    ) -> float:
+        return super().__new__(cls, val)
+
+    def __init__(self, val: Union[str, float, Decimal],
+                 validation_mode: int = None) -> None:
+        # If a string passed, then store it
+        if isinstance(val, str):
+            self.original_string = val.strip()
+        elif isinstance(val, (IS, ISfloat)) and hasattr(val, 'original_string'):
+            self.original_string = val.original_string
+        if validation_mode:
+            msg = (
+                f'Value "{str(self)}" is not valid for elements with a VR '
+                'of IS'
+            )
+            if validation_mode == config.WARN:
+                warnings.warn(msg)
+            elif validation_mode == config.RAISE:
+                msg += (
+                    "\nSet reading_validation_mode to WARN or IGNORE to bypass"
+                )
+                raise TypeError(msg)
+
+
 class IS(int):
     """Store value for an element with VR **IS** as :class:`int`.
 
@@ -1258,7 +1300,7 @@ class IS(int):
     def __new__(  # type: ignore[misc]
             cls: Type["IS"], val: Union[None, str, int, float, Decimal],
             validation_mode: int = None
-    ) -> Optional[Union[str, "IS"]]:
+    ) -> Optional[Union[str, "IS", "ISfloat"]]:
         """Create instance if new integer string"""
         if val is None:
             return val
@@ -1272,16 +1314,16 @@ def __new__(  # type: ignore[misc]
             validate_value("IS", val, validation_mode)
 
         try:
-            newval = super().__new__(cls, val)
+            newval: Union[IS, ISfloat] = super().__new__(cls, val)
         except ValueError:
             # accept float strings when no integer loss, e.g. "1.0"
             newval = super().__new__(cls, float(val))
 
-        # check if a float or Decimal passed in, then could have lost info,
-        # and will raise error. E.g. IS(Decimal('1')) is ok, but not IS(1.23)
-        #   IS('1.23') will raise ValueError
+        # If a float or Decimal was passed in, check for non-integer,
+        # i.e. could lose info if converted to int
+        # If so, create an ISfloat instead (if allowed by settings)
         if isinstance(val, (float, Decimal, str)) and newval != float(val):
-            raise TypeError("Could not convert value to integer without loss")
+            newval = ISfloat(val, validation_mode)
 
         # Checks in case underlying int is >32 bits, DICOM does not allow this
         if (not -2**31 <= newval < 2**31 and


Test patch:
diff --git a/pydicom/tests/test_json.py b/pydicom/tests/test_json.py
--- a/pydicom/tests/test_json.py
+++ b/pydicom/tests/test_json.py
@@ -5,6 +5,7 @@
 
 import pytest
 
+from pydicom import config
 from pydicom import dcmread
 from pydicom.data import get_testdata_file
 from pydicom.dataelem import DataElement, RawDataElement
@@ -293,13 +294,10 @@ def test_suppress_invalid_tags_with_failed_dataelement(self):
         # we have to add a RawDataElement as creating a DataElement would
         # already raise an exception
         ds[0x00082128] = RawDataElement(
-            Tag(0x00082128), 'IS', 4, b'5.25', 0, True, True)
-
-        with pytest.raises(TypeError):
-            ds.to_json_dict()
+            Tag(0x00082128), 'IS', 4, b'5.25', 0, True, True
+        )
 
         ds_json = ds.to_json_dict(suppress_invalid_tags=True)
-
         assert "00082128" not in ds_json
 
 
diff --git a/pydicom/tests/test_valuerep.py b/pydicom/tests/test_valuerep.py
--- a/pydicom/tests/test_valuerep.py
+++ b/pydicom/tests/test_valuerep.py
@@ -6,6 +6,7 @@
 from datetime import datetime, date, time, timedelta, timezone
 from decimal import Decimal
 from itertools import chain
+from io import BytesIO
 import pickle
 import math
 import sys
@@ -19,9 +20,10 @@
 from pydicom.data import get_testdata_file
 from pydicom.dataset import Dataset
 from pydicom._dicom_dict import DicomDictionary, RepeatersDictionary
+from pydicom.filereader import read_dataset
 from pydicom.tag import Tag
 from pydicom.valuerep import (
-    DS, IS, DSfloat, DSdecimal, PersonName, VR, STANDARD_VR,
+    DS, IS, DSfloat, DSdecimal, ISfloat, PersonName, VR, STANDARD_VR,
     AMBIGUOUS_VR, STR_VR, BYTES_VR, FLOAT_VR, INT_VR, LIST_VR
 )
 from pydicom.values import convert_value
@@ -889,11 +891,31 @@ def test_valid_value(self, disable_value_validation):
         assert 42 == IS("42.0")
         assert 42 == IS(42.0)
 
+    def test_float_value(self):
+        """Read binary value of IS that is actually a float"""
+        # from issue #1661
+        # Create BytesIO with single data element for Exposure (0018,1152)
+        #   length 4, value "14.5"
+        bin_elem = b"\x18\x00\x52\x11\x04\x00\x00\x0014.5"
+        with BytesIO(bin_elem) as bio:
+            ds = read_dataset(bio, True, True)
+        assert isinstance(ds.Exposure, ISfloat)
+        assert ds.Exposure == 14.5
+
+        # Strict checking raises an error
+        with pytest.raises(ValueError):
+            _ = IS("14.5", validation_mode=config.RAISE)
+        with pytest.raises(TypeError):
+            _ = IS(14.5, validation_mode=config.RAISE)
+
+    def test_float_init(self):
+        """New ISfloat created from another behaves correctly"""
+        is1 = IS("14.5", validation_mode=config.IGNORE)
+        is2 = IS(is1)
+        assert is1 == is2
+        assert is2.original_string == is1.original_string
+
     def test_invalid_value(self, disable_value_validation):
-        with pytest.raises(TypeError, match="Could not convert value"):
-            IS(0.9)
-        with pytest.raises(TypeError, match="Could not convert value"):
-            IS("0.9")
         with pytest.raises(ValueError, match="could not convert string"):
             IS("foo")