Repository: pydicom/pydicom

Issue:
Handling of DS too long to be encoded in explicit encoding
<!-- Instructions For Filing a Bug: https://github.com/pydicom/pydicom/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
This is probably not a bug, but I'm not sure about the wanted behavior.
An RTPlan dataset encoded as Little Endian Implicit contains multiple values in the DS tag DHV Data (3004,0058) with an overall length not fitting into 2 bytes. Trying to write this as explicit Little Endian fails with an exception (`"ushort format requires 0 &lt;= number &lt;= (0x7fff * 2 + 1)"`) which is raised by the `pack` call in `write_leUS` while trying to write the length.

The standard says for this case in PS3.5, Table 6.2-1 (for VR DS):
```
Note
Data Elements with multiple values using this VR may not be properly encoded if Explicit-VR transfer syntax is used and the VL of this attribute exceeds 65534 bytes.
```
So, as I understand it, this is valid DICOM, that cannot be converted to explicit encoding without data loss.
The question is how to handle this. What comes to mind:
- truncate the value and log a warning
- raise a meaningful exception
- adapt the behavior depending on some config setting

Any thoughts?

<!-- Example: Attribute Error thrown when printing (0x0010, 0x0020) patient Id> 0-->

#### Steps/Code to Reproduce
<!--
Example:
```py
from io import BytesIO
from pydicom import dcmread

bytestream = b'\x02\x00\x02\x00\x55\x49\x16\x00\x31\x2e\x32\x2e\x38\x34\x30\x2e\x31' \
             b'\x30\x30\x30\x38\x2e\x35\x2e\x31\x2e\x31\x2e\x39\x00\x02\x00\x10\x00' \
             b'\x55\x49\x12\x00\x31\x2e\x32\x2e\x38\x34\x30\x2e\x31\x30\x30\x30\x38' \
             b'\x2e\x31\x2e\x32\x00\x20\x20\x10\x00\x02\x00\x00\x00\x01\x00\x20\x20' \
             b'\x20\x00\x06\x00\x00\x00\x4e\x4f\x52\x4d\x41\x4c'

fp = BytesIO(bytestream)
ds = dcmread(fp, force=True)

print(ds.PatientID)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com

When possible use pydicom testing examples to reproduce the errors. Otherwise, provide
an anonymous version of the data in order to replicate the errors.
-->

#### Expected Results
<!-- Please paste or describe the expected results.
Example: No error is thrown and the name of the patient is printed.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback.
(Use %xmode to deactivate ipython's trace beautifier)
Example: ```AttributeError: 'FileDataset' object has no attribute 'PatientID'```
-->

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import pydicom; print("pydicom", pydicom.__version__)
-->


<!-- Thanks for contributing! -->



Discussion:
Interesting situation.  To summarize:  with implicit VR and the four-byte length it uses, we can read in something longer than **Ex**plicit VR can write, because it uses only two bytes for encoding the length (for VRs like DS).

I say we catch the exception and re-raise with a meaningful error message.  If the user wants to truncate the data to avoid this, they can choose how to do so, i.e. what to leave out.  Or they can save the dataset using Implicit VR.  The error message could explain those two options.


> I say we catch the exception and re-raise with a meaningful error message

Agreed - I will have a go at this some time later. 
I have to admit that I was quite surprised to see this behavior - while I did know that implicit transfer syntax has other length fields, I never thought that this would ever matter.
Reopen as the fix is incorrect - the VR shall be changed to UN instead (see #900).

Patch:
diff --git a/pydicom/filewriter.py b/pydicom/filewriter.py
--- a/pydicom/filewriter.py
+++ b/pydicom/filewriter.py
@@ -3,7 +3,7 @@
 
 from __future__ import absolute_import
 
-import struct
+import warnings
 from struct import pack
 
 from pydicom import compat
@@ -439,27 +439,19 @@ def write_data_element(fp, data_element, encodings=None):
     # Write element's tag
     fp.write_tag(data_element.tag)
 
-    # If explicit VR, write the VR
-    VR = data_element.VR
-    if not fp.is_implicit_VR:
-        if len(VR) != 2:
-            msg = ("Cannot write ambiguous VR of '{}' for data element with "
-                   "tag {}.\nSet the correct VR before writing, or use an "
-                   "implicit VR transfer syntax".format(
-                       VR, repr(data_element.tag)))
-            raise ValueError(msg)
-        if not in_py2:
-            fp.write(bytes(VR, default_encoding))
-        else:
-            fp.write(VR)
-        if VR in extra_length_VRs:
-            fp.write_US(0)  # reserved 2 bytes
-
     # write into a buffer to avoid seeking back which can be expansive
     buffer = DicomBytesIO()
     buffer.is_little_endian = fp.is_little_endian
     buffer.is_implicit_VR = fp.is_implicit_VR
 
+    VR = data_element.VR
+    if not fp.is_implicit_VR and len(VR) != 2:
+        msg = ("Cannot write ambiguous VR of '{}' for data element with "
+               "tag {}.\nSet the correct VR before writing, or use an "
+               "implicit VR transfer syntax".format(
+                   VR, repr(data_element.tag)))
+        raise ValueError(msg)
+
     if data_element.is_raw:
         # raw data element values can be written as they are
         buffer.write(data_element.value)
@@ -496,18 +488,29 @@ def write_data_element(fp, data_element, encodings=None):
                              'start with an item tag')
 
     value_length = buffer.tell()
+    if (not fp.is_implicit_VR and VR not in extra_length_VRs and
+            not is_undefined_length and value_length > 0xffff):
+        # see PS 3.5, section 6.2.2 for handling of this case
+        msg = ('The value for the data element {} exceeds the size '
+               'of 64 kByte and cannot be written in an explicit transfer '
+               'syntax. The data element VR is changed from "{}" to "UN" '
+               'to allow saving the data.'
+               .format(data_element.tag, VR))
+        warnings.warn(msg)
+        VR = 'UN'
+
+    # write the VR for explicit transfer syntax
+    if not fp.is_implicit_VR:
+        if not in_py2:
+            fp.write(bytes(VR, default_encoding))
+        else:
+            fp.write(VR)
+        if VR in extra_length_VRs:
+            fp.write_US(0)  # reserved 2 bytes
+
     if (not fp.is_implicit_VR and VR not in extra_length_VRs and
             not is_undefined_length):
-        try:
-            fp.write_US(value_length)  # Explicit VR length field is 2 bytes
-        except struct.error:
-            msg = ('The value for the data element {} exceeds the size '
-                   'of 64 kByte and cannot be written in an explicit transfer '
-                   'syntax. You can save it using Implicit Little Endian '
-                   'transfer syntax, or you have to truncate the value to not '
-                   'exceed the maximum size of 64 kByte.'
-                   .format(data_element.tag))
-            raise ValueError(msg)
+        fp.write_US(value_length)  # Explicit VR length field is 2 bytes
     else:
         # write the proper length of the data_element in the length slot,
         # unless is SQ with undefined length.


Test patch:
diff --git a/pydicom/tests/test_filewriter.py b/pydicom/tests/test_filewriter.py
--- a/pydicom/tests/test_filewriter.py
+++ b/pydicom/tests/test_filewriter.py
@@ -2513,20 +2513,31 @@ def test_writing_too_big_data_in_explicit_encoding(self):
                                  large_value,
                                  is_undefined_length=False)
         write_data_element(self.fp, pixel_data)
+        self.fp.seek(0)
+        ds = read_dataset(self.fp, True, True)
+        assert 'DS' == ds[0x30040058].VR
 
         self.fp = DicomBytesIO()
         self.fp.is_little_endian = True
         self.fp.is_implicit_VR = False
-        # shall raise if trying to write it with explicit transfer syntax,
-        # where the length field is 2 bytes long
-        expected_message = (r'The value for the data element \(3004, 0058\) '
-                            r'exceeds the size of 64 kByte ')
-        with pytest.raises(ValueError, match=expected_message):
+
+        msg = (r'The value for the data element \(3004, 0058\) exceeds the '
+               r'size of 64 kByte and cannot be written in an explicit '
+               r'transfer syntax. The data element VR is changed from '
+               r'"DS" to "UN" to allow saving the data.')
+
+        with pytest.warns(UserWarning, match=msg):
             write_data_element(self.fp, pixel_data)
+        self.fp.seek(0)
+        ds = read_dataset(self.fp, False, True)
+        assert 'UN' == ds[0x30040058].VR
 
+        # we expect the same behavior in Big Endian transfer syntax
         self.fp = DicomBytesIO()
         self.fp.is_little_endian = False
         self.fp.is_implicit_VR = False
-        # we expect the same behavior in Big Endian transfer syntax
-        with pytest.raises(ValueError, match=expected_message):
+        with pytest.warns(UserWarning, match=msg):
             write_data_element(self.fp, pixel_data)
+        self.fp.seek(0)
+        ds = read_dataset(self.fp, False, False)
+        assert 'UN' == ds[0x30040058].VR