Repository: pydicom/pydicom

Issue:
Encoding to ISO 2022 IR 159 doesn't work
<!-- Instructions For Filing a Bug: https://github.com/pydicom/pydicom/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
Encoding to ISO 2022 IR 159 doesn't work even if 'ISO 2022 IR 159' is passed to pydicom.charset.convert_encodings.

#### Steps/Code to Reproduce
ISO 2022 IR 159 is designed as supplement characters to ISO 2022 IR 87. So these characters are not frequent use. But person name sometimes contains them. In the following example, the letter of "鷗" is only in ISO 2022 IR 159. But we cannot encode them correctly. 

```
import pydicom

japanese_pn = u"Mori^Ogai=森^鷗外=もり^おうがい"
specific_character_sets = ["ISO 2022 IR 6", "ISO 2022 IR 87", "ISO 2022 IR 159"]
expect_encoded = (
    b"\x4d\x6f\x72\x69\x5e\x4f\x67\x61\x69\x3d\x1b\x24\x42\x3f"
    b"\x39\x1b\x28\x42\x5e\x1b\x24\x28\x44\x6c\x3f\x1b\x24\x42"
    b"\x33\x30\x1b\x28\x42\x3d\x1b\x24\x42\x24\x62\x24\x6a\x1b"
    b"\x28\x42\x5e\x1b\x24\x42\x24\x2a\x24\x26\x24\x2c\x24\x24"
    b"\x1b\x28\x42"
)

python_encodings = pydicom.charset.convert_encodings(specific_character_sets)
actual_encoded = pydicom.charset.encode_string(japanese_pn, python_encodings)

print("actual:{}".format(actual_encoded))
print("expect:{}".format(expect_encoded))
```
#### Expected Results
<!-- Please paste or describe the expected results.
Example: No error is thrown and the name of the patient is printed.-->
```
b'Mori^Ogai=\x1b$B?9\x1b(B^\x1b$(Dl?\x1b$B30\x1b(B=\x1b$B$b$j\x1b(B^\x1b$B$*$&$,$$\x1b(B'
```
#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback.
(Use %xmode to deactivate ipython's trace beautifier)
Example: ```AttributeError: 'FileDataset' object has no attribute 'PatientID'```
-->
```
b'Mori^Ogai=?^??=??^????'
```

And the followin exception occurs.

```
/PATH/TO/MY/PYTHON/PACKAGES/pydicom/charset.py:488: UserWarning: Failed to encode value with encodings: iso8859, iso2022_jp, iso-2022-jp - using replacement characters in encoded string
  .format(', '.join(encodings)))
```

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import pydicom; print("pydicom", pydicom.__version__)
-->
```
Linux-4.15.0-55-generic-x86_64-with-debian-buster-sid
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0]
pydicom 1.3.0
```


<!-- Thanks for contributing! -->



Discussion:
In my opinion, this is caused by selecting the wrong python codec corresponding to 'ISO 2022 IR 159'. In the current implementation,  'iso-2022-jp' is used if 'ISO 2022 IR 159' is passed. But 'iso-2022-jp' is alias to 'iso200_jp'. I think we have to use 'iso-2022-jp-2'. It contains all 'iso-2022-jp' characters and 'ISO 2022 IR 159' characters.

If you don't mind, please assign this issue to me. I will make a PR for this issue.
Sure, go ahead! And thanks for the support!

Patch:
diff --git a/pydicom/charset.py b/pydicom/charset.py
--- a/pydicom/charset.py
+++ b/pydicom/charset.py
@@ -45,7 +45,7 @@
     'ISO 2022 IR 144': 'iso_ir_144',
     'ISO 2022 IR 148': 'iso_ir_148',
     'ISO 2022 IR 149': 'euc_kr',
-    'ISO 2022 IR 159': 'iso-2022-jp',
+    'ISO 2022 IR 159': 'iso2022_jp_2',
     'ISO 2022 IR 166': 'iso_ir_166',
     'ISO 2022 IR 58': 'iso_ir_58',
     'ISO_IR 192': 'UTF8',  # from Chinese example, 2008 PS3.5 Annex J p1-4
@@ -81,7 +81,7 @@
     ESC + b'-M': 'iso_ir_148',
     ESC + b'-T': 'iso_ir_166',
     ESC + b'$)C': 'euc_kr',
-    ESC + b'$(D': 'iso-2022-jp',
+    ESC + b'$(D': 'iso2022_jp_2',
     ESC + b'$)A': 'iso_ir_58',
 }
 
@@ -92,7 +92,7 @@
 # To decode them, the escape sequence shall be preserved in the input byte
 # string, and will be removed during decoding by Python.
 handled_encodings = ('iso2022_jp',
-                     'iso-2022-jp',
+                     'iso2022_jp_2',
                      'iso_ir_58')
 
 
@@ -165,10 +165,18 @@ def _encode_to_jis_x_0201(value, errors='strict'):
 
     return encoded
 
-
 def _encode_to_jis_x_0208(value, errors='strict'):
-    """Convert a unicode string into JIS X 0208 byte string using iso2022_jp
-    encodings.
+    """Convert a unicode string into JIS X 0208 byte string."""
+    return _encode_to_given_charset(value, 'ISO 2022 IR 87', errors=errors)
+
+
+def _encode_to_jis_x_0212(value, errors='strict'):
+    """Convert a unicode string into JIS X 0212 byte string."""
+    return _encode_to_given_charset(value, 'ISO 2022 IR 159', errors=errors)
+
+
+def _encode_to_given_charset(value, character_set, errors='strict'):
+    """Convert a unicode string into given character set.
     The escape sequence which is located at the end of the encoded value has
     to vary depending on the value 1 of SpecificCharacterSet. So we have to
     trim it and append the correct escape sequence manually.
@@ -177,6 +185,8 @@ def _encode_to_jis_x_0208(value, errors='strict'):
     ----------
     value : text type
         The unicode string as presented to the user.
+    character_set: str:
+        Character set for result.
     errors : str
         The behavior of a character which could not be encoded. This value
         is passed to errors argument of str.encode().
@@ -185,28 +195,30 @@ def _encode_to_jis_x_0208(value, errors='strict'):
     -------
     byte string
         The encoded string. If some characters in value could not be encoded to
-        JIS X 0208, it depends on the behavior of iso2022_jp encoder.
+        given character_set, it depends on the behavior of corresponding python
+        encoder.
 
     Raises
     ------
     UnicodeEncodeError
         If errors is set to 'strict' and `value` could not be encoded with
-        JIS X 0208.
+        given character_set.
     """
 
+    encoding = python_encoding[character_set]
     # If errors is not strict, this function is used as fallback.
     # So keep the tail escape sequence of encoded for backward compatibility.
     if errors != 'strict':
-        return value.encode('iso2022_jp', errors=errors)
+        return value.encode(encoding, errors=errors)
 
-    Encoder = codecs.getincrementalencoder('iso2022-jp')
+    Encoder = codecs.getincrementalencoder(encoding)
     encoder = Encoder()
 
     encoded = encoder.encode(value[0])
-    if encoded[:3] != ENCODINGS_TO_CODES['iso2022_jp']:
+    if not encoded.startswith(ENCODINGS_TO_CODES[encoding]):
         raise UnicodeEncodeError(
-            'iso2022_jp', value, 0, len(value),
-            'Given character is out of ISO IR 87')
+            encoding, value, 0, len(value),
+            'Given character is out of {}'.format(character_set))
 
     for i, c in enumerate(value[1:], 1):
         try:
@@ -215,10 +227,10 @@ def _encode_to_jis_x_0208(value, errors='strict'):
             e.start = i
             e.end = len(value)
             raise e
-        if b[:3] == ENCODINGS_TO_CODES['iso8859']:
+        if b[:1] == ESC:
             raise UnicodeEncodeError(
-                'iso2022_jp', value, i, len(value),
-                'Given character is out of ISO IR 87')
+                encoding, value, i, len(value),
+                'Given character is out of {}'.format(character_set))
         encoded += b
     return encoded
 
@@ -262,13 +274,13 @@ def _get_escape_sequence_for_encoding(encoding, encoded=None):
 
 
 # These encodings need escape sequence to handle alphanumeric characters.
-need_tail_escape_sequence_encodings = ('iso2022_jp', 'iso-2022-jp')
+need_tail_escape_sequence_encodings = ('iso2022_jp', 'iso2022_jp_2')
 
 
 custom_encoders = {
     'shift_jis': _encode_to_jis_x_0201,
     'iso2022_jp': _encode_to_jis_x_0208,
-    'iso-2022-jp': _encode_to_jis_x_0208
+    'iso2022_jp_2': _encode_to_jis_x_0212
 }
 
 
@@ -562,7 +574,7 @@ def _encode_string_parts(value, encodings):
     # unencoded_part is empty - we are done, return the encoded string
     if best_encoding in need_tail_escape_sequence_encodings:
         encoded += _get_escape_sequence_for_encoding(encodings[0])
-    return encoded
+    return bytes(encoded)
 
 
 def _encode_string_impl(value, encoding, errors='strict'):


Test patch:
diff --git a/pydicom/tests/test_charset.py b/pydicom/tests/test_charset.py
--- a/pydicom/tests/test_charset.py
+++ b/pydicom/tests/test_charset.py
@@ -9,6 +9,7 @@
 from pydicom.data import get_charset_files, get_testdata_files
 from pydicom.dataelem import DataElement
 from pydicom.filebase import DicomBytesIO
+from pydicom.valuerep import PersonName3
 
 # The file names (without '.dcm' extension) of most of the character test
 # files, together with the respective decoded PatientName tag values.
@@ -197,9 +198,9 @@ def test_bad_decoded_multi_byte_encoding(self):
                            b'\x1b$(D\xc4\xe9\xef\xed\xf5\xf3\xe9\xef\xf2')
 
         with pytest.warns(UserWarning, match='Failed to decode byte string '
-                                             'with encodings: iso-2022-jp'):
+                                             'with encodings: iso2022_jp_2'):
             pydicom.charset.decode_element(elem, ['ISO 2022 IR 159'])
-            assert u'����������' == elem.value
+            assert u'���������' == elem.value
 
     def test_bad_decoded_multi_byte_encoding_enforce_standard(self):
         """Test handling bad encoding for single encoding if
@@ -207,7 +208,7 @@ def test_bad_decoded_multi_byte_encoding_enforce_standard(self):
         config.enforce_valid_values = True
         elem = DataElement(0x00100010, 'PN',
                            b'\x1b$(D\xc4\xe9\xef\xed\xf5\xf3\xe9\xef\xf2')
-        msg = ("'iso2022_jp' codec can't decode bytes in position 0-3: "
+        msg = ("'iso2022_jp_2' codec can't decode byte 0xc4 in position 4: "
                "illegal multibyte sequence")
         with pytest.raises(UnicodeDecodeError, match=msg):
             pydicom.charset.decode_element(elem, ['ISO 2022 IR 159'])
@@ -435,11 +436,26 @@ def test_japanese_multi_byte_personname(self):
             ds_out = dcmread(fp)
             assert original_string == ds_out.PatientName.original_string
 
+        japanese_pn = PersonName3(u"Mori^Ogai=森^鷗外=もり^おうがい")
+        pyencs = pydicom.charset.convert_encodings(["ISO 2022 IR 6",
+                                                    "ISO 2022 IR 87",
+                                                    "ISO 2022 IR 159"])
+        actual_encoded = bytes(japanese_pn.encode(pyencs))
+        expect_encoded = (
+            b"\x4d\x6f\x72\x69\x5e\x4f\x67\x61\x69\x3d\x1b\x24\x42\x3f"
+            b"\x39\x1b\x28\x42\x5e\x1b\x24\x28\x44\x6c\x3f\x1b\x24\x42"
+            b"\x33\x30\x1b\x28\x42\x3d\x1b\x24\x42\x24\x62\x24\x6a\x1b"
+            b"\x28\x42\x5e\x1b\x24\x42\x24\x2a\x24\x26\x24\x2c\x24\x24"
+            b"\x1b\x28\x42"
+        )
+        assert expect_encoded == actual_encoded
+
     def test_japanese_multi_byte_encoding(self):
         """Test japanese multi byte strings are correctly encoded."""
-        encoded = pydicom.charset.encode_string(u'あaｱア',
-                                                ['shift_jis', 'iso2022_jp'])
-        assert b'\x1b$B$"\x1b(Ja\x1b)I\xb1\x1b$B%"\x1b(J' == encoded
+        encoded = pydicom.charset.encode_string(u'あaｱア齩', ['shift_jis',
+                                                'iso2022_jp', 'iso2022_jp_2'])
+        expect = b'\x1b$B$"\x1b(Ja\x1b)I\xb1\x1b$B%"\x1b$(DmN\x1b(J'
+        assert expect == bytes(encoded)
 
     def test_bad_japanese_encoding(self):
         """Test japanese multi byte strings are not correctly encoded."""