Repository: pydicom/pydicom

Issue:
to_json does not work with binary data in pixel_array
**Describe the issue**
Loading a dicom file and then performing a to_json() on it does not work with binary data in pixel_array.



**Expected behavior**
I would have expected that a base64 conversion is first performed on the binary data and then encoded to json. 

**Steps To Reproduce**
How to reproduce the issue. Please include:
1. A minimum working code sample

import pydicom
ds = pydicom.dcmread('path_to_file')
output = ds.to_json()


2. The traceback (if one occurred)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.virtualenvs/my_env/lib/python3.7/site-packages/pydicom/dataset.py", line 2003, in to_json
    dump_handler=dump_handler
  File "/.virtualenvs/my_env/lib/python3.7/site-packages/pydicom/dataset.py", line 1889, in _data_element_to_json
    binary_value = data_element.value.encode('utf-8')
AttributeError: 'bytes' object has no attribute 'encode'


3. Which of the following packages are available and their versions:
  * Numpy
numpy==1.17.2
  * Pillow
Pillow==6.1.0
  * JPEG-LS
  * GDCM
4. The anonymized DICOM dataset (if possible).

**Your environment**
Please run the following and paste the output.
```bash
$ python -c "import platform; print(platform.platform())"
Darwin-19.2.0-x86_64-i386-64bit
$ python -c "import sys; print('Python ', sys.version)"
Python  3.7.6 (default, Dec 30 2019, 19:38:26) 
[Clang 11.0.0 (clang-1100.0.33.16)]
$ python -c "import pydicom; print('pydicom ', pydicom.__version__)"
pydicom  1.3.0
```



Discussion:
Can you please check with pydicom 1.4? Binary data handling should have been fixed there. 
ok works now once I set the bulk_data_threshold value to a higher value. 

Thank you!
Ok, that may be an issue with the data size. Currently, the default for `bulk_data_threshold` is 1, if I remember correctly, which may not be the best value - meaning that all binary data larger than that expect a bulk data element handler. Setting the threshold to a large value shall fix this, if the data is encoded directly.
Ah, you already found that, ok!

yep. thanks, works now. 
@darcymason - json support is still flagged as alpha - I think we can consider it at least beta now, and add a small section in the documentation.
We may also rethink the `bulk_data_threshold` parameter - maybe just ignore it if no bulk data handler is set, and set the default value to some sensible value (1kB or something).
> @darcymason - json support is still flagged as alpha - I think we can consider it at least beta now, and add a small section in the documentation.
> We may also rethink the `bulk_data_threshold` parameter - maybe just ignore it if no bulk data handler is set, and set the default value to some sensible value (1kB or something).

I agree that updating the documentation is a good idea for this. As its pretty common to have binary image data in dicom files, and its guaranteed to fail with the default value for bulk_data_threshold

Thanks once again though!
Ping @pieper, @hackermd for comment about `bulk_data_threshold`.
> We may also rethink the bulk_data_threshold parameter - maybe just ignore it if no bulk data handler is set, and set the default value to some sensible value (1kB or something).

1k threshold makes sense to me.  👍 

Patch:
diff --git a/pydicom/dataelem.py b/pydicom/dataelem.py
--- a/pydicom/dataelem.py
+++ b/pydicom/dataelem.py
@@ -315,32 +315,21 @@ def to_json_dict(self, bulk_data_element_handler, bulk_data_threshold):
             via DICOMweb WADO-RS
         bulk_data_threshold: int
             Size of base64 encoded data element above which a value will be
-            provided in form of a "BulkDataURI" rather than "InlineBinary"
+            provided in form of a "BulkDataURI" rather than "InlineBinary".
+            Ignored if no bulk data handler is given.
 
         Returns
         -------
         dict
             Mapping representing a JSON encoded data element
-
-        Raises
-        ------
-        TypeError
-            When size of encoded data element exceeds `bulk_data_threshold`
-            but `bulk_data_element_handler` is ``None`` and hence not callable
-
         """
         json_element = {'vr': self.VR, }
         if self.VR in jsonrep.BINARY_VR_VALUES:
             if not self.is_empty:
                 binary_value = self.value
                 encoded_value = base64.b64encode(binary_value).decode('utf-8')
-                if len(encoded_value) > bulk_data_threshold:
-                    if bulk_data_element_handler is None:
-                        raise TypeError(
-                            'No bulk data element handler provided to '
-                            'generate URL for value of data element "{}".'
-                            .format(self.name)
-                        )
+                if (bulk_data_element_handler is not None and
+                        len(encoded_value) > bulk_data_threshold):
                     json_element['BulkDataURI'] = bulk_data_element_handler(
                         self
                     )
@@ -398,7 +387,7 @@ def to_json_dict(self, bulk_data_element_handler, bulk_data_threshold):
             )
         return json_element
 
-    def to_json(self, bulk_data_threshold=1, bulk_data_element_handler=None,
+    def to_json(self, bulk_data_threshold=1024, bulk_data_element_handler=None,
                 dump_handler=None):
         """Return a JSON representation of the :class:`DataElement`.
 
@@ -412,7 +401,8 @@ def to_json(self, bulk_data_threshold=1, bulk_data_element_handler=None,
             via DICOMweb WADO-RS
         bulk_data_threshold: int
             Size of base64 encoded data element above which a value will be
-            provided in form of a "BulkDataURI" rather than "InlineBinary"
+            provided in form of a "BulkDataURI" rather than "InlineBinary".
+            Ignored if no bulk data handler is given.
         dump_handler : callable, optional
             Callable function that accepts a :class:`dict` and returns the
             serialized (dumped) JSON string (by default uses
@@ -423,12 +413,6 @@ def to_json(self, bulk_data_threshold=1, bulk_data_element_handler=None,
         dict
             Mapping representing a JSON encoded data element
 
-        Raises
-        ------
-        TypeError
-            When size of encoded data element exceeds `bulk_data_threshold`
-            but `bulk_data_element_handler` is ``None`` and hence not callable
-
         See also
         --------
         Dataset.to_json
diff --git a/pydicom/dataset.py b/pydicom/dataset.py
--- a/pydicom/dataset.py
+++ b/pydicom/dataset.py
@@ -2126,11 +2126,11 @@ def from_json(cls, json_dataset, bulk_data_uri_handler=None):
             dataset.add(data_element)
         return dataset
 
-    def to_json_dict(self, bulk_data_threshold=1,
+    def to_json_dict(self, bulk_data_threshold=1024,
                      bulk_data_element_handler=None):
         """Return a dictionary representation of the :class:`Dataset`
         conforming to the DICOM JSON Model as described in the DICOM
-        Standard, Part 18, :dcm:`Annex F<part18/chaptr_F.html>`.
+        Standard, Part 18, :dcm:`Annex F<part18/chapter_F.html>`.
 
         .. versionadded:: 1.4
 
@@ -2140,7 +2140,7 @@ def to_json_dict(self, bulk_data_threshold=1,
             Threshold for the length of a base64-encoded binary data element
             above which the element should be considered bulk data and the
             value provided as a URI rather than included inline (default:
-            ``1``).
+            ``1024``). Ignored if no bulk data handler is given.
         bulk_data_element_handler : callable, optional
             Callable function that accepts a bulk data element and returns a
             JSON representation of the data element (dictionary including the
@@ -2161,7 +2161,7 @@ def to_json_dict(self, bulk_data_threshold=1,
             )
         return json_dataset
 
-    def to_json(self, bulk_data_threshold=1, bulk_data_element_handler=None,
+    def to_json(self, bulk_data_threshold=1024, bulk_data_element_handler=None,
                 dump_handler=None):
         """Return a JSON representation of the :class:`Dataset`.
 
@@ -2175,7 +2175,7 @@ def to_json(self, bulk_data_threshold=1, bulk_data_element_handler=None,
             Threshold for the length of a base64-encoded binary data element
             above which the element should be considered bulk data and the
             value provided as a URI rather than included inline (default:
-            ``1``).
+            ``1024``). Ignored if no bulk data handler is given.
         bulk_data_element_handler : callable, optional
             Callable function that accepts a bulk data element and returns a
             JSON representation of the data element (dictionary including the


Test patch:
diff --git a/pydicom/tests/test_json.py b/pydicom/tests/test_json.py
--- a/pydicom/tests/test_json.py
+++ b/pydicom/tests/test_json.py
@@ -1,12 +1,11 @@
 # -*- coding: utf-8 -*-
 # Copyright 2008-2019 pydicom authors. See LICENSE file for details.
 import json
-import sys
 
 import pytest
 
 from pydicom import dcmread, compat
-from pydicom.data import get_testdata_files
+from pydicom.data import get_testdata_file
 from pydicom.dataelem import DataElement
 from pydicom.dataset import Dataset
 from pydicom.tag import Tag, BaseTag
@@ -15,7 +14,7 @@
 
 class TestPersonName(object):
     def test_json_pn_from_file(self):
-        with open(get_testdata_files("test_PN.json")[0]) as s:
+        with open(get_testdata_file("test_PN.json")) as s:
             ds = Dataset.from_json(s.read())
         assert isinstance(ds[0x00080090].value,
                           (PersonNameUnicode, PersonName3))
@@ -144,7 +143,7 @@ def test_from_json(self):
         assert 0x000910AF == ds[0x00091001].value
         assert [0x00100010, 0x00100020, 0x00100030] == ds[0x00091002].value
 
-    def test_invalid_json(self):
+    def test_invalid_value_in_json(self):
         ds_json = ('{"00091001": {"vr": "AT", "Value": ["000910AG"]}, '
                    '"00091002": {"vr": "AT", "Value": ["00100010"]}}')
         with pytest.warns(UserWarning, match='Invalid value "000910AG" for '
@@ -153,15 +152,24 @@ def test_invalid_json(self):
             assert ds[0x00091001].value is None
             assert 0x00100010 == ds[0x00091002].value
 
+    def test_invalid_tag_in_json(self):
+        ds_json = ('{"000910AG": {"vr": "AT", "Value": ["00091000"]}, '
+                   '"00091002": {"vr": "AT", "Value": ["00100010"]}}')
+        with pytest.raises(ValueError, match='Data element "000910AG" could '
+                                             'not be loaded from JSON:'):
+            ds = Dataset.from_json(ds_json)
+            assert ds[0x00091001].value is None
+            assert 0x00100010 == ds[0x00091002].value
+
 
 class TestDataSetToJson(object):
     def test_json_from_dicom_file(self):
-        ds1 = Dataset(dcmread(get_testdata_files("CT_small.dcm")[0]))
-        ds_json = ds1.to_json(bulk_data_threshold=100000)
+        ds1 = Dataset(dcmread(get_testdata_file("CT_small.dcm")))
+        ds_json = ds1.to_json()
         ds2 = Dataset.from_json(ds_json)
         assert ds1 == ds2
 
-        ds_json = ds1.to_json_dict(bulk_data_threshold=100000)
+        ds_json = ds1.to_json_dict()
         ds2 = Dataset.from_json(ds_json)
         assert ds1 == ds2
 
@@ -205,7 +213,7 @@ def test_roundtrip(self):
         ds.add_new(0x00091101, 'SH', 'Version2')
         ds.add_new(0x00091102, 'US', 2)
 
-        json_string = ds.to_json(bulk_data_threshold=100)
+        json_string = ds.to_json()
         json_model = json.loads(json_string)
 
         assert json_model['00080005']['Value'] == ['ISO_IR 100']
@@ -223,7 +231,7 @@ def test_roundtrip(self):
         ds2 = Dataset.from_json(json_model)
         assert ds == ds2
 
-        json_model2 = ds.to_json_dict(bulk_data_threshold=100)
+        json_model2 = ds.to_json_dict()
         if compat.in_py2:
             # in Python 2, the encoding of this is slightly different
             # (single vs double quotation marks)
@@ -274,7 +282,7 @@ def test_sort_order(self):
 
 class TestSequence(object):
     def test_nested_sequences(self):
-        test1_json = get_testdata_files("test1.json")[0]
+        test1_json = get_testdata_file("test1.json")
         with open(test1_json) as f:
             with pytest.warns(UserWarning,
                               match='no bulk data URI handler provided '):
@@ -288,7 +296,7 @@ class TestBinary(object):
     def test_inline_binary(self):
         ds = Dataset()
         ds.add_new(0x00091002, 'OB', b'BinaryContent')
-        ds_json = ds.to_json_dict(bulk_data_threshold=20)
+        ds_json = ds.to_json_dict()
         assert "00091002" in ds_json
         assert "QmluYXJ5Q29udGVudA==" == ds_json["00091002"]["InlineBinary"]
         ds1 = Dataset.from_json(ds_json)