Repository: pydicom/pydicom

Issue:
Strings with Value Representation DS are too long
**Describe the bug**
Strings of Value Representation DS are restricted to a maximum length of 16 bytes according to [Part 5 Section 6.2](http://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html#para_15754884-9ca2-4b12-9368-d66f32bc8ce1), but `pydicom.valuerep.DS` may represent numbers with more than 16 bytes.

**Expected behavior**
`pydicom.valuerep.DS` should create a string of maximum length 16, when passed a fixed point number with many decimals.

**Steps To Reproduce**
```python
len(str(pydicom.valuerep.DS(3.14159265358979323846264338327950288419716939937510582097)).encode('utf-8'))
len(str(pydicom.valuerep.DS("3.14159265358979323846264338327950288419716939937510582097")).encode('utf-8'))
```
returns `17` and `58`, respectively, instead of `16`.

**Your environment**
```
module       | version
------       | -------
platform     | macOS-10.15.6-x86_64-i386-64bit
Python       | 3.8.6 (default, Oct  8 2020, 14:06:32)  [Clang 12.0.0 (clang-1200.0.32.2)]
pydicom      | 2.0.0
gdcm         | _module not found_
jpeg_ls      | _module not found_
numpy        | 1.19.4
PIL          | 8.0.1
```


Discussion:
Thanks for this, @hackermd.  Pydicom has traditionally been permissive about values as they are set, because sometimes people want to replicate existing invalid DICOM.  But for sure this should be an error when `config.enforce_valid_values` is `True` (perhaps a warning otherwise), and pydicom should leave it to the calling code to figure out how to truncate the value.
I've just checked, because I thought I remembered this coming up before.  There is a check - but only for `DS` derived from `Decimal` (and when `enforce_valid_values is True`).  Wouldn't be hard to replicate the checks for the `DSfloat` class.

Patch:
diff --git a/pydicom/valuerep.py b/pydicom/valuerep.py
--- a/pydicom/valuerep.py
+++ b/pydicom/valuerep.py
@@ -3,6 +3,7 @@
 
 import datetime
 from decimal import Decimal
+from math import floor, isfinite, log10
 import platform
 import re
 import sys
@@ -329,15 +330,139 @@ def __reduce_ex__(self, protocol: int) -> Union[str, Tuple[Any, ...]]:
             )
 
 
+# Regex to match strings that represent valid DICOM decimal strings (DS)
+_DS_REGEX = re.compile(r'\s*[\+\-]?\d+(\.\d+)?([eE][\+\-]?\d+)?\s*$')
+
+
+def is_valid_ds(s: str) -> bool:
+    """Check whether this string is a valid decimal string.
+
+    Valid decimal strings must be 16 characters or fewer, and contain only
+    characters from a limited set.
+
+    Parameters
+    ----------
+    s: str
+        String to test.
+
+    Returns
+    -------
+    bool
+        True if the string is a valid decimal string. Otherwise False.
+    """
+    # Check that the length is within the limits
+    if len(s) > 16:
+        return False
+
+    return _DS_REGEX.match(s) is not None
+
+
+def format_number_as_ds(val: Union[float, Decimal]) -> str:
+    """Truncate a float's representation to give a valid Decimal String (DS).
+
+    DICOM's decimal string (DS) representation is limited to strings with 16
+    characters and a limited set of characters. This function represents a
+    float that satisfies these constraints while retaining as much
+    precision as possible. Some floats are represented using scientific
+    notation to make more efficient use of the limited number of characters.
+
+    Note that this will incur a loss of precision if the number cannot be
+    represented with 16 characters. Furthermore, non-finite floats (infs and
+    nans) cannot be represented as decimal strings and will cause an error to
+    be raised.
+
+    Parameters
+    ----------
+    val: Union[float, Decimal]
+        The floating point value whose representation is required.
+
+    Returns
+    -------
+    str
+        String representation of the float satisfying the constraints of the
+        decimal string representation.
+
+    Raises
+    ------
+    ValueError
+        If val does not represent a finite value
+
+    """
+    if not isinstance(val, (float, Decimal)):
+        raise TypeError("'val' must be of type float or decimal.Decimal")
+    if not isfinite(val):
+        raise ValueError(
+            "Cannot encode non-finite floats as DICOM decimal strings. "
+            f"Got '{val}'"
+        )
+
+    valstr = str(val)
+
+    # In the simple case, the default python string representation
+    # will do
+    if len(valstr) <= 16:
+        return valstr
+
+    # Decide whether to use scientific notation
+    logval = log10(abs(val))
+
+    # Characters needed for '-' at start
+    sign_chars = 1 if val < 0.0 else 0
+
+    # Numbers larger than 1e14 cannot be correctly represented by truncating
+    # their string representations to 16 chars, e.g pi * 10^13 would become
+    # '314159265358979.', which may not be universally understood. This limit
+    # is 1e13 for negative numbers because of the minus sign.
+    # For negative exponents, the point of equal precision between scientific
+    # and standard notation is 1e-4 e.g. '0.00031415926535' and
+    # '3.1415926535e-04' are both 16 chars
+    use_scientific = logval < -4 or logval >= (14 - sign_chars)
+
+    if use_scientific:
+        # In principle, we could have a number where the exponent
+        # needs three digits to be represented (bigger than this cannot be
+        # represented by floats). Due to floating point limitations
+        # this is best checked for by doing the string conversion
+        remaining_chars = 10 - sign_chars
+        trunc_str = f'%.{remaining_chars}e' % val
+        if len(trunc_str) > 16:
+            trunc_str = f'%.{remaining_chars - 1}e' % val
+        return trunc_str
+    else:
+        if logval >= 1.0:
+            # chars remaining for digits after sign, digits left of '.' and '.'
+            remaining_chars = 14 - sign_chars - int(floor(logval))
+        else:
+            remaining_chars = 14 - sign_chars
+        return f'%.{remaining_chars}f' % val
+
+
 class DSfloat(float):
     """Store value for an element with VR **DS** as :class:`float`.
 
     If constructed from an empty string, return the empty string,
     not an instance of this class.
 
+    Parameters
+    ----------
+    val: Union[str, int, float, Decimal]
+        Value to store as a DS.
+    auto_format: bool
+        If True, automatically format the string representation of this
+        number to ensure it satisfies the constraints in the DICOM standard.
+        Note that this will lead to loss of precision for some numbers.
+
     """
+    def __new__(
+            cls,
+            val: Union[str, int, float, Decimal],
+            auto_format: bool = False
+    ) -> [_DSfloat]:
+        return super().__new__(cls, val)
+
     def __init__(
-        self, val: Union[str, int, float, Decimal]
+        self, val: Union[str, int, float, Decimal],
+        auto_format: bool = False
     ) -> None:
         """Store the original string if one given, for exact write-out of same
         value later.
@@ -350,28 +475,70 @@ def __init__(
         elif isinstance(val, (DSfloat, DSdecimal)) and has_attribute:
             self.original_string = val.original_string
 
+        self.auto_format = auto_format
+        if self.auto_format:
+            # If auto_format is True, keep the float value the same, but change
+            # the string representation stored in original_string if necessary
+            if hasattr(self, 'original_string'):
+                if not is_valid_ds(self.original_string):
+                    self.original_string = format_number_as_ds(
+                        float(self.original_string)
+                    )
+            else:
+                self.original_string = format_number_as_ds(self)
+
+        if config.enforce_valid_values and not self.auto_format:
+            if len(repr(self).strip('"')) > 16:
+                raise OverflowError(
+                    "Values for elements with a VR of 'DS' must be <= 16 "
+                    "characters long, but the float provided requires > 16 "
+                    "characters to be accurately represented. Use a smaller "
+                    "string, set 'config.enforce_valid_values' to False to "
+                    "override the length check, or explicitly construct a DS "
+                    "object with 'auto_format' set to True"
+                )
+            if not is_valid_ds(repr(self).strip('"')):
+                # This will catch nan and inf
+                raise ValueError(
+                    f'Value "{str(self)}" is not valid for elements with a VR '
+                    'of DS'
+                )
+
     def __str__(self) -> str:
-        if hasattr(self, 'original_string'):
+        if hasattr(self, 'original_string') and not self.auto_format:
             return self.original_string
 
         # Issue #937 (Python 3.8 compatibility)
         return repr(self)[1:-1]
 
     def __repr__(self) -> str:
+        if self.auto_format and hasattr(self, 'original_string'):
+            return f'"{self.original_string}"'
         return f'"{super().__repr__()}"'
 
 
 class DSdecimal(Decimal):
     """Store value for an element with VR **DS** as :class:`decimal.Decimal`.
 
+    Parameters
+    ----------
+    val: Union[str, int, float, Decimal]
+        Value to store as a DS.
+    auto_format: bool
+        If True, automatically format the string representation of this
+        number to ensure it satisfies the constraints in the DICOM standard.
+        Note that this will lead to loss of precision for some numbers.
+
     Notes
     -----
     If constructed from an empty string, returns the empty string, not an
     instance of this class.
+
     """
     def __new__(
         cls: Type[_DSdecimal],
-        val: Union[str, int, float, Decimal]
+        val: Union[str, int, float, Decimal],
+        auto_format: bool = False
     ) -> Optional[_DSdecimal]:
         """Create an instance of DS object, or return a blank string if one is
         passed in, e.g. from a type 2 DICOM blank value.
@@ -395,19 +562,13 @@ def __new__(
                 return None
 
         val = super().__new__(cls, val)
-        if len(str(val)) > 16 and config.enforce_valid_values:
-            raise OverflowError(
-                "Values for elements with a VR of 'DS' values must be <= 16 "
-                "characters long. Use a smaller string, set "
-                "'config.enforce_valid_values' to False to override the "
-                "length check, or use 'Decimal.quantize()' and initialize "
-                "with a 'Decimal' instance."
-            )
 
         return val
 
     def __init__(
-        self, val: Union[str, int, float, Decimal]
+        self,
+        val: Union[str, int, float, Decimal],
+        auto_format: bool = False
     ) -> None:
         """Store the original string if one given, for exact write-out of same
         value later. E.g. if set ``'1.23e2'``, :class:`~decimal.Decimal` would
@@ -421,6 +582,35 @@ def __init__(
         elif isinstance(val, (DSfloat, DSdecimal)) and has_str:
             self.original_string = val.original_string
 
+        self.auto_format = auto_format
+        if self.auto_format:
+            # If auto_format is True, keep the float value the same, but change
+            # the string representation stored in original_string if necessary
+            if hasattr(self, 'original_string'):
+                if not is_valid_ds(self.original_string):
+                    self.original_string = format_number_as_ds(
+                        float(self.original_string)
+                    )
+            else:
+                self.original_string = format_number_as_ds(self)
+
+        if config.enforce_valid_values:
+            if len(repr(self).strip('"')) > 16:
+                raise OverflowError(
+                    "Values for elements with a VR of 'DS' values must be "
+                    "<= 16 characters long. Use a smaller string, set "
+                    "'config.enforce_valid_values' to False to override the "
+                    "length check, use 'Decimal.quantize()' and initialize "
+                    "with a 'Decimal' instance, or explicitly construct a DS "
+                    "instance with 'auto_format' set to True"
+                )
+            if not is_valid_ds(repr(self).strip('"')):
+                # This will catch nan and inf
+                raise ValueError(
+                    f'Value "{str(self)}" is not valid for elements with a VR '
+                    'of DS'
+                )
+
     def __str__(self) -> str:
         has_str = hasattr(self, 'original_string')
         if has_str and len(self.original_string) <= 16:
@@ -429,6 +619,8 @@ def __str__(self) -> str:
         return super().__str__()
 
     def __repr__(self) -> str:
+        if self.auto_format and hasattr(self, 'original_string'):
+            return f'"{self.original_string}"'
         return f'"{str(self)}"'
 
 
@@ -440,7 +632,8 @@ def __repr__(self) -> str:
 
 
 def DS(
-    val: Union[None, str, int, float, Decimal]
+    val: Union[None, str, int, float, Decimal],
+    auto_format: bool = False
 ) -> Union[None, str, DSfloat, DSdecimal]:
     """Factory function for creating DS class instances.
 
@@ -458,7 +651,7 @@ def DS(
     if val == '' or val is None:
         return val
 
-    return DSclass(val)
+    return DSclass(val, auto_format=auto_format)
 
 
 class IS(int):


Test patch:
diff --git a/pydicom/tests/test_valuerep.py b/pydicom/tests/test_valuerep.py
--- a/pydicom/tests/test_valuerep.py
+++ b/pydicom/tests/test_valuerep.py
@@ -9,7 +9,9 @@
     import cPickle as pickle
 except ImportError:
     import pickle
+import math
 import sys
+from typing import Union
 
 from pydicom.tag import Tag
 from pydicom.values import convert_value
@@ -29,6 +31,27 @@
 default_encoding = "iso8859"
 
 
+@pytest.fixture()
+def enforce_valid_true_fixture():
+    """Fixture to run tests with enforce_valid_values True and ensure it is
+       reset afterwards regardless of whether test succeeds."""
+    enforce_flag_original = config.enforce_valid_values
+    config.enforce_valid_values = True
+    yield
+    config.enforce_valid_values = enforce_flag_original
+
+
+@pytest.fixture(params=(True, False))
+def enforce_valid_both_fixture(request):
+    """Fixture to run tests with enforce_valid_values with both True and False
+       and ensure it is reset afterwards regardless of whether test succeeds.
+    """
+    enforce_flag_original = config.enforce_valid_values
+    config.enforce_valid_values = request.param
+    yield
+    config.enforce_valid_values = enforce_flag_original
+
+
 class TestTM:
     """Unit tests for pickling TM"""
     def test_pickling(self):
@@ -191,6 +214,115 @@ def test_new_obj_conversion(self):
             pydicom.valuerep.DA(123456)
 
 
+class TestIsValidDS:
+    """Unit tests for the is_valid_ds function."""
+    @pytest.mark.parametrize(
+        's',
+        [
+            '1',
+            '3.14159265358979',
+            '-1234.456e78',
+            '1.234E-5',
+            '1.234E+5',
+            '+1',
+            '    42',  # leading spaces allowed
+            '42    ',  # trailing spaces allowed
+        ]
+    )
+    def test_valid(self, s: str):
+        """Various valid decimal strings."""
+        assert pydicom.valuerep.is_valid_ds(s)
+
+    @pytest.mark.parametrize(
+        's',
+        [
+            'nan',
+            '-inf',
+            '3.141592653589793',  # too long
+            '1,000',              # no commas
+            '1 000',              # no embedded spaces
+            '127.0.0.1',          # not a number
+            '1.e',                # not a number
+            '',
+        ]
+    )
+    def test_invalid(self, s: str):
+        """Various invalid decimal strings."""
+        assert not pydicom.valuerep.is_valid_ds(s)
+
+
+class TestTruncateFloatForDS:
+    """Unit tests for float truncation function"""
+    def check_valid(self, s: str) -> bool:
+        # Use the pydicom test function
+        if not pydicom.valuerep.is_valid_ds(s):
+            return False
+
+        # Disallow floats ending in '.' since this may not be correctly
+        # interpreted
+        if s.endswith('.'):
+            return False
+
+        # Otherwise return True
+        return True
+
+    @pytest.mark.parametrize(
+        'val,expected_str',
+        [
+            [1.0, "1.0"],
+            [0.0, "0.0"],
+            [-0.0, "-0.0"],
+            [0.123, "0.123"],
+            [-0.321, "-0.321"],
+            [0.00001, "1e-05"],
+            [3.14159265358979323846, '3.14159265358979'],
+            [-3.14159265358979323846, '-3.1415926535898'],
+            [5.3859401928763739403e-7, '5.3859401929e-07'],
+            [-5.3859401928763739403e-7, '-5.385940193e-07'],
+            [1.2342534378125532912998323e10, '12342534378.1255'],
+            [6.40708699858767842501238e13, '64070869985876.8'],
+            [1.7976931348623157e+308, '1.797693135e+308'],
+        ]
+    )
+    def test_auto_format(self, val: float, expected_str: str):
+        """Test truncation of some basic values."""
+        assert pydicom.valuerep.format_number_as_ds(val) == expected_str
+
+    @pytest.mark.parametrize(
+        'exp', [-101, -100, 100, 101] + list(range(-16, 17))
+    )
+    def test_powers_of_pi(self, exp: int):
+        """Raise pi to various powers to test truncation."""
+        val = math.pi * 10 ** exp
+        s = pydicom.valuerep.format_number_as_ds(val)
+        assert self.check_valid(s)
+
+    @pytest.mark.parametrize(
+        'exp', [-101, -100, 100, 101] + list(range(-16, 17))
+    )
+    def test_powers_of_negative_pi(self, exp: int):
+        """Raise negative pi to various powers to test truncation."""
+        val = -math.pi * 10 ** exp
+        s = pydicom.valuerep.format_number_as_ds(val)
+        assert self.check_valid(s)
+
+    @pytest.mark.parametrize(
+        'val', [float('-nan'), float('nan'), float('-inf'), float('inf')]
+    )
+    def test_invalid(self, val: float):
+        """Test non-finite floating point numbers raise an error"""
+        with pytest.raises(ValueError):
+            pydicom.valuerep.format_number_as_ds(val)
+
+    def test_wrong_type(self):
+        """Test calling with a string raises an error"""
+        with pytest.raises(
+            TypeError,
+            match="'val' must be of type float or decimal.Decimal"
+        ):
+            pydicom.valuerep.format_number_as_ds('1.0')
+
+
 class TestDS:
     """Unit tests for DS values"""
     def test_empty_value(self):
@@ -249,6 +381,57 @@ def test_DSdecimal(self):
         assert 1.2345 == y
         assert "1.2345" == y.original_string
 
+    def test_auto_format(self, enforce_valid_both_fixture):
+        """Test truncating floats"""
+        x = pydicom.valuerep.DSfloat(math.pi, auto_format=True)
+
+        # Float representation should be unaltered by truncation
+        assert x == math.pi
+        # String representations should be correctly formatted
+        assert str(x) == '3.14159265358979'
+        assert repr(x) == '"3.14159265358979"'
+
+    def test_auto_format_invalid_string(self, enforce_valid_both_fixture):
+        """If the user supplies an invalid string, this should be formatted."""
+        x = pydicom.valuerep.DSfloat('3.141592653589793', auto_format=True)
+
+        # Float representation should be unaltered by truncation
+        assert x == float('3.141592653589793')
+        # String representations should be correctly formatted
+        assert str(x) == '3.14159265358979'
+        assert repr(x) == '"3.14159265358979"'
+
+    def test_auto_format_valid_string(self, enforce_valid_both_fixture):
+        """If the user supplies a valid string, this should not be altered."""
+        x = pydicom.valuerep.DSfloat('1.234e-1', auto_format=True)
+
+        # Float representation should be correct
+        assert x == 0.1234
+        # String representations should be unaltered
+        assert str(x) == '1.234e-1'
+        assert repr(x) == '"1.234e-1"'
+
+    def test_enforce_valid_values_length(self, enforce_valid_true_fixture):
+        """Test that errors are raised when length is too long."""
+        with pytest.raises(OverflowError):
+            valuerep.DSfloat('3.141592653589793')
+
+    @pytest.mark.parametrize(
+        'val',
+        [
+            'nan', '-nan', 'inf', '-inf', float('nan'), float('-nan'),
+            float('-inf'), float('inf')
+        ]
+    )
+    def test_enforce_valid_values_value(
+        self,
+        val: Union[float, str],
+        enforce_valid_true_fixture
+    ):
+        """Test that errors are raised when value is invalid."""
+        with pytest.raises(ValueError):
+            valuerep.DSfloat(val)
+
 
 class TestDSdecimal:
     """Unit tests for pickling DSdecimal"""
@@ -297,6 +480,52 @@ def test_repr(self):
         x = pydicom.valuerep.DSdecimal('1.2345')
         assert '"1.2345"' == repr(x)
 
+    def test_auto_format(self, enforce_valid_both_fixture):
+        """Test truncating decimal"""
+        x = pydicom.valuerep.DSdecimal(Decimal(math.pi), auto_format=True)
+
+        # Decimal representation should be unaltered by truncation
+        assert x == Decimal(math.pi)
+        # String representations should be correctly formatted
+        assert str(x) == '3.14159265358979'
+        assert repr(x) == '"3.14159265358979"'
+
+    def test_auto_format_invalid_string(self, enforce_valid_both_fixture):
+        """If the user supplies an invalid string, this should be formatted."""
+        x = pydicom.valuerep.DSdecimal('3.141592653589793', auto_format=True)
+
+        # Decimal representation should be unaltered by truncation
+        assert x == Decimal('3.141592653589793')
+        # String representations should be correctly formatted
+        assert str(x) == '3.14159265358979'
+        assert repr(x) == '"3.14159265358979"'
+
+    @pytest.mark.parametrize(
+        'val',
+        [
+            'NaN', '-NaN', 'Infinity', '-Infinity', Decimal('NaN'),
+            Decimal('-NaN'), Decimal('-Infinity'), Decimal('Infinity')
+        ]
+    )
+    def test_enforce_valid_values_value(
+        self,
+        val: Union[Decimal, str],
+        enforce_valid_true_fixture
+    ):
+        """Test that errors are raised when value is invalid."""
+        with pytest.raises(ValueError):
+            valuerep.DSdecimal(val)
+
+    def test_auto_format_valid_string(self, enforce_valid_both_fixture):
+        """If the user supplies a valid string, this should not be altered."""
+        x = pydicom.valuerep.DSdecimal('1.234e-1', auto_format=True)
+
+        # Decimal representation should be correct
+        assert x == Decimal('1.234e-1')
+        # String representations should be unaltered
+        assert str(x) == '1.234e-1'
+        assert repr(x) == '"1.234e-1"'
+
 
 class TestIS:
     """Unit tests for IS"""
@@ -355,7 +584,6 @@ def test_str(self):
         val = pydicom.valuerep.IS("1.0")
         assert "1.0" == str(val)
 
-
     def test_repr(self):
         """Test IS.__repr__()."""
         val = pydicom.valuerep.IS(1)
@@ -428,13 +656,6 @@ def test_valid_decimal_strings(self):
         assert isinstance(ds, valuerep.DSdecimal)
         assert len(str(ds)) <= 16
 
-        # Now the input string is too long but decimal.Decimal can convert it
-        # to a valid 16-character string
-        long_str = "-0.000000981338674"
-        ds = valuerep.DS(long_str)
-        assert isinstance(ds, valuerep.DSdecimal)
-        assert len(str(ds)) <= 16
-
     def test_invalid_decimal_strings(self, enforce_valid_values):
         # Now the input string truly is invalid
         invalid_string = "-9.813386743e-006"