Repository: pydicom/pydicom

Issue:
Wrong encoding occurs if the value 1 of SpecificCharacterSets is ISO 2022 IR 13.
<!-- Instructions For Filing a Bug: https://github.com/pydicom/pydicom/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
All Japanese characters are encoded into shift_jis if the value 1 of SpecificCharacterSets (0x0008, 0x0005) is  ISO 2022 IR 13.

#### Steps/Code to Reproduce
The japanese_pn and expect_encoded in the following code came from 
[H.3.2 Value 1 of Attribute Specific Character Set (0008,0005) is ISO 2022 IR 13.](http://dicom.nema.org/medical/dicom/2015b/output/chtml/part05/sect_H.3.2.html)

```py
import pydicom

japanese_pn = u"ﾔﾏﾀﾞ^ﾀﾛｳ=山田^太郎=やまだ^たろう"
specific_character_sets = ["ISO 2022 IR 13", "ISO 2022 IR 87"]
expect_encoded = (
    b"\xd4\xcf\xc0\xde\x5e\xc0\xdb\xb3\x3d\x1b\x24\x42\x3b\x33"
    b"\x45\x44\x1b\x28\x4a\x5e\x1b\x24\x42\x42\x40\x4f\x3a\x1b"
    b"\x28\x4a\x3d\x1b\x24\x42\x24\x64\x24\x5e\x24\x40\x1b\x28"
    b"\x4a\x5e\x1b\x24\x42\x24\x3f\x24\x6d\x24\x26\x1b\x28\x4a"
)

python_encodings = pydicom.charset.convert_encodings(specific_character_sets)
actual_encoded = pydicom.charset.encode_string(japanese_pn, python_encodings)

print("actual:{}".format(actual_encoded))
print("expect:{}".format(expect_encoded))
```
<!--
Example:
```py
from io import BytesIO
from pydicom import dcmread

bytestream = b'\x02\x00\x02\x00\x55\x49\x16\x00\x31\x2e\x32\x2e\x38\x34\x30\x2e\x31' \
             b'\x30\x30\x30\x38\x2e\x35\x2e\x31\x2e\x31\x2e\x39\x00\x02\x00\x10\x00' \
             b'\x55\x49\x12\x00\x31\x2e\x32\x2e\x38\x34\x30\x2e\x31\x30\x30\x30\x38' \
             b'\x2e\x31\x2e\x32\x00\x20\x20\x10\x00\x02\x00\x00\x00\x01\x00\x20\x20' \
             b'\x20\x00\x06\x00\x00\x00\x4e\x4f\x52\x4d\x41\x4c'

fp = BytesIO(bytestream)
ds = dcmread(fp, force=True)

print(ds.PatientID)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com

When possible use pydicom testing examples to reproduce the errors. Otherwise, provide
an anonymous version of the data in order to replicate the errors.
-->

#### Expected Results
<!-- Please paste or describe the expected results.
Example: No error is thrown and the name of the patient is printed.-->
```
b'\xd4\xcf\xc0\xde^\xc0\xdb\xb3=\x1b$B;3ED\x1b(J^\x1b$BB@O:\x1b(J=\x1b$B$d$^$@\x1b(J^\x1b$B$?$m$&\x1b(J'
```

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback.
(Use %xmode to deactivate ipython's trace beautifier)
Example: ```AttributeError: 'FileDataset' object has no attribute 'PatientID'```
-->
```
b'\xd4\xcf\xc0\xde^\xc0\xdb\xb3=\x8eR\x93c^\x91\xbe\x98Y=\x82\xe2\x82\xdc\x82\xbe^\x82\xbd\x82\xeb\x82\xa4'
```

#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import pydicom; print("pydicom", pydicom.__version__)
-->
```
Linux-4.15.0-50-generic-x86_64-with-debian-buster-sid
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0]
pydicom 1.2.2
```

<!-- Thanks for contributing! -->



Discussion:
In my opinion, this issue is caused by encoding to shift_jis doesn't raise UnicodeError when characters which are out of JIS X 0201 are given.  So I guess that this is fixed by encoding to jis correctly.

If you don't mind, please assign this issue to me. I will make a PR  for this issue.
Thanks for the report - of course you can make a PR for this, please go ahead! 
Dear all.
I'm trying to solve this issue. And I want some advice about the scope of this issue and the way of implementation.
May I discuss them in this issue thread? Or should I create a PR and add W.I.P to its title?
Whatever suits you better - if you want to discuss concrete code, it may be easier to add a PR to be able to comment on specific lines, but that's completely up to you!
@mrbean-bremen 

Thank for your quick reply. I got it. First, I will write some concrete codes. And then I'll make a PR and want to discuss there. 

Patch:
diff --git a/pydicom/charset.py b/pydicom/charset.py
--- a/pydicom/charset.py
+++ b/pydicom/charset.py
@@ -86,6 +86,7 @@
 }
 
 ENCODINGS_TO_CODES = {v: k for k, v in CODES_TO_ENCODINGS.items()}
+ENCODINGS_TO_CODES['shift_jis'] = ESC + b')I'
 
 # Multi-byte character sets except Korean are handled by Python.
 # To decode them, the escape sequence shall be preserved in the input byte
@@ -95,6 +96,182 @@
                      'iso_ir_58')
 
 
+def _encode_to_jis_x_0201(value, errors='strict'):
+    """Convert a unicode string into JIS X 0201 byte string using shift_jis
+    encodings.
+    shift_jis is a superset of jis_x_0201. So we can regard the encoded value
+    as jis_x_0201 if it is single byte character.
+
+    Parameters
+    ----------
+    value : text type
+        The unicode string as presented to the user.
+    errors : str
+        The behavior of a character which could not be encoded. If 'strict' is
+        passed, raise an UnicodeEncodeError. If any other value is passed,
+        non ISO IR 14 characters are replaced by the ASCII '?'.
+
+    Returns
+    -------
+    byte string
+        The encoded string. If some characters in value could not be encoded to
+        JIS X 0201, and `errors` is not set to 'strict', they are replaced to
+        '?'.
+
+    Raises
+    ------
+    UnicodeEncodeError
+        If errors is set to 'strict' and `value` could not be encoded with
+        JIS X 0201.
+    """
+
+    Encoder = codecs.getincrementalencoder('shift_jis')
+    encoder = Encoder()
+
+    # If errors is not strict, this function is used as fallback.
+    # In this case, we use only ISO IR 14 to encode given value
+    # without escape sequence.
+    if errors != 'strict' or value == '':
+        encoded = b''
+        for c in value:
+            try:
+                b = encoder.encode(c)
+            except UnicodeEncodeError as e:
+                b = b'?'
+
+            if len(b) != 1 or 0x80 <= ord(b):
+                b = b'?'
+            encoded += b
+        return encoded
+
+    encoded = encoder.encode(value[0])
+    if len(encoded) != 1:
+        raise UnicodeEncodeError(
+            'shift_jis', value, 0, len(value), 'illegal multibyte sequence')
+
+    msb = ord(encoded) & 0x80  # msb is 1 for ISO IR 13, 0 for ISO IR 14
+    for i, c in enumerate(value[1:], 1):
+        try:
+            b = encoder.encode(c)
+        except UnicodeEncodeError as e:
+            e.start = i
+            e.end = len(value)
+            raise e
+        if len(b) != 1 or ((ord(b) & 0x80) ^ msb) != 0:
+            character_set = 'ISO IR 14' if msb == 0 else 'ISO IR 13'
+            msg = 'Given character is out of {}'.format(character_set)
+            raise UnicodeEncodeError('shift_jis', value, i, len(value), msg)
+        encoded += b
+
+    return encoded
+
+
+def _encode_to_jis_x_0208(value, errors='strict'):
+    """Convert a unicode string into JIS X 0208 byte string using iso2022_jp
+    encodings.
+    The escape sequence which is located at the end of the encoded value has
+    to vary depending on the value 1 of SpecificCharacterSet. So we have to
+    trim it and append the correct escape sequence manually.
+
+    Parameters
+    ----------
+    value : text type
+        The unicode string as presented to the user.
+    errors : str
+        The behavior of a character which could not be encoded. This value
+        is passed to errors argument of str.encode().
+
+    Returns
+    -------
+    byte string
+        The encoded string. If some characters in value could not be encoded to
+        JIS X 0208, it depends on the behavior of iso2022_jp encoder.
+
+    Raises
+    ------
+    UnicodeEncodeError
+        If errors is set to 'strict' and `value` could not be encoded with
+        JIS X 0208.
+    """
+
+    # If errors is not strict, this function is used as fallback.
+    # So keep the tail escape sequence of encoded for backward compatibility.
+    if errors != 'strict':
+        return value.encode('iso2022_jp', errors=errors)
+
+    Encoder = codecs.getincrementalencoder('iso2022-jp')
+    encoder = Encoder()
+
+    encoded = encoder.encode(value[0])
+    if encoded[:3] != ENCODINGS_TO_CODES['iso2022_jp']:
+        raise UnicodeEncodeError(
+            'iso2022_jp', value, 0, len(value),
+            'Given character is out of ISO IR 87')
+
+    for i, c in enumerate(value[1:], 1):
+        try:
+            b = encoder.encode(c)
+        except UnicodeEncodeError as e:
+            e.start = i
+            e.end = len(value)
+            raise e
+        if b[:3] == ENCODINGS_TO_CODES['iso8859']:
+            raise UnicodeEncodeError(
+                'iso2022_jp', value, i, len(value),
+                'Given character is out of ISO IR 87')
+        encoded += b
+    return encoded
+
+
+def _get_escape_sequence_for_encoding(encoding, encoded=None):
+    """ Return an escape sequence corresponding to the given encoding. If
+    encoding is 'shift_jis', return 'ESC)I' or 'ESC(J' depending on the first
+    byte of encoded.
+
+    Parameters
+    ----------
+    encoding : str
+        An encoding is used to specify  an escape sequence.
+
+    encoded : bytes or str
+        The encoded value is used to chose an escape sequence if encoding is
+        'shift_jis'
+
+    Returns
+    -------
+    string
+        Escape sequence for encoded value.
+    """
+
+    ESC_ISO_IR_14 = ESC + b'(J'
+    ESC_ISO_IR_13 = ESC + b')I'
+
+    if encoding == 'shift_jis':
+        if encoded is None:
+            return ESC_ISO_IR_14
+
+        if not in_py2:
+            first_byte = encoded[0]
+        else:
+            first_byte = ord(encoded[0])
+        if 0x80 <= first_byte:
+            return ESC_ISO_IR_13
+
+        return ESC_ISO_IR_14
+    return ENCODINGS_TO_CODES.get(encoding, b'')
+
+
+# These encodings need escape sequence to handle alphanumeric characters.
+need_tail_escape_sequence_encodings = ('iso2022_jp', 'iso-2022-jp')
+
+
+custom_encoders = {
+    'shift_jis': _encode_to_jis_x_0201,
+    'iso2022_jp': _encode_to_jis_x_0208,
+    'iso-2022-jp': _encode_to_jis_x_0208
+}
+
+
 def decode_string(value, encodings, delimiters):
     """Convert a raw byte string into a unicode string using the given
     list of encodings.
@@ -281,9 +458,14 @@ def encode_string(value, encodings):
     """
     for i, encoding in enumerate(encodings):
         try:
-            encoded = value.encode(encoding)
+            encoded = _encode_string_impl(value, encoding)
+
             if i > 0 and encoding not in handled_encodings:
-                return ENCODINGS_TO_CODES.get(encoding, b'') + encoded
+                escape_sequence = _get_escape_sequence_for_encoding(
+                        encoding, encoded=encoded)
+                encoded = escape_sequence + encoded
+            if encoding in need_tail_escape_sequence_encodings:
+                encoded += _get_escape_sequence_for_encoding(encodings[0])
             return encoded
         except UnicodeError:
             continue
@@ -304,7 +486,7 @@ def encode_string(value, encodings):
         warnings.warn("Failed to encode value with encodings: {} - using "
                       "replacement characters in encoded string"
                       .format(', '.join(encodings)))
-        return value.encode(encodings[0], errors='replace')
+        return _encode_string_impl(value, encodings[0], errors='replace')
 
 
 def _encode_string_parts(value, encodings):
@@ -337,14 +519,14 @@ def _encode_string_parts(value, encodings):
     """
     encoded = bytearray()
     unencoded_part = value
+    best_encoding = None
     while unencoded_part:
         # find the encoding that can encode the longest part of the rest
         # of the string still to be encoded
         max_index = 0
-        best_encoding = None
         for encoding in encodings:
             try:
-                unencoded_part.encode(encoding)
+                _encode_string_impl(unencoded_part, encoding)
                 # if we get here, the whole rest of the value can be encoded
                 best_encoding = encoding
                 max_index = len(unencoded_part)
@@ -355,20 +537,36 @@ def _encode_string_parts(value, encodings):
                     max_index = e.start
                     best_encoding = encoding
         # none of the given encodings can encode the first character - give up
-        if best_encoding is None:
-            raise ValueError()
+        if max_index == 0:
+            raise ValueError("None of the given encodings can encode the "
+                             "first character")
 
         # encode the part that can be encoded with the found encoding
-        encoded_part = unencoded_part[:max_index].encode(best_encoding)
+        encoded_part = _encode_string_impl(unencoded_part[:max_index],
+                                           best_encoding)
         if best_encoding not in handled_encodings:
-            encoded += ENCODINGS_TO_CODES.get(best_encoding, b'')
+            encoded += _get_escape_sequence_for_encoding(
+                    best_encoding, encoded=encoded_part)
         encoded += encoded_part
         # set remaining unencoded part of the string and handle that
         unencoded_part = unencoded_part[max_index:]
     # unencoded_part is empty - we are done, return the encoded string
+    if best_encoding in need_tail_escape_sequence_encodings:
+        encoded += _get_escape_sequence_for_encoding(encodings[0])
     return encoded
 
 
+def _encode_string_impl(value, encoding, errors='strict'):
+    """Convert a unicode string into a byte string. If given encoding is in
+    custom_encoders, use a corresponding custom_encoder. If given encoding
+    is not in custom_encoders, use a corresponding python handled encoder.
+    """
+    if encoding in custom_encoders:
+        return custom_encoders[encoding](value, errors=errors)
+    else:
+        return value.encode(encoding, errors=errors)
+
+
 # DICOM PS3.5-2008 6.1.1 (p 18) says:
 #   default is ISO-IR 6 G0, equiv to common chr set of ISO 8859 (PS3.5 6.1.2.1)
 #    (0008,0005)  value 1 can *replace* the default encoding...


Test patch:
diff --git a/pydicom/tests/test_charset.py b/pydicom/tests/test_charset.py
--- a/pydicom/tests/test_charset.py
+++ b/pydicom/tests/test_charset.py
@@ -404,3 +404,36 @@ def test_invalid_second_encoding(self):
                                 u"- using default encoding instead"):
             pydicom.charset.decode(
                 elem, ['ISO 2022 IR 100', 'ISO 2022 IR 146'])
+
+    def test_japanese_multi_byte_personname(self):
+        """Test japanese person name which has multi byte strings are
+        correctly encoded."""
+        file_path = get_charset_files('chrH32.dcm')[0]
+        ds = dcmread(file_path)
+        ds.decode()
+
+        if hasattr(ds.PatientName, 'original_string'):
+            original_string = ds.PatientName.original_string
+            ds.PatientName.original_string = None
+            fp = DicomBytesIO()
+            fp.is_implicit_VR = False
+            fp.is_little_endian = True
+            ds.save_as(fp, write_like_original=False)
+            fp.seek(0)
+            ds_out = dcmread(fp)
+            assert original_string == ds_out.PatientName.original_string
+
+    def test_japanese_multi_byte_encoding(self):
+        """Test japanese multi byte strings are correctly encoded."""
+        encoded = pydicom.charset.encode_string(u'あaｱア',
+                                                ['shift_jis', 'iso2022_jp'])
+        assert b'\x1b$B$"\x1b(Ja\x1b)I\xb1\x1b$B%"\x1b(J' == encoded
+
+    def test_bad_japanese_encoding(self):
+        """Test japanese multi byte strings are not correctly encoded."""
+        with pytest.warns(UserWarning,
+                          match=u"Failed to encode value with encodings"
+                                u": shift_jis - using replacement character"
+                                u"s in encoded string"):
+            encoded = pydicom.charset.encode_string(u'あaｱア', ['shift_jis'])
+            assert b'?a??' == encoded