Repository: pydicom/pydicom

Issue:
Some `DataElement`s have huge strs
#### Description
For some VRs, the `maxBytesToDisplay` value is not used to trim `repval`, resulting in a `DataElement` with a huge `str` representation.

#### Steps/Code to Reproduce
Example:
```py
from pydicom import Dataset

ds = Dataset()
ds.TwoDimensionalToThreeDimensionalMapData = b"\0"*1024*1024
print(ds)
```
or
```py
from pydicom import Dataset

ds = Dataset()
ds.TwoDimensionalToThreeDimensionalMapData = [1.0]*1024*256
print(ds)
```

#### Expected Results
A short message, something like "Array of 1048576 elements" or "Array of 262144 elements", depending if the parsing for floats is done or not.

#### Actual Results
The whole 1MB bytearray / 256K-float-aray is printed.

#### Versions
Python 3.6.5
pydicom 1.1.0

Discussion:
pep8speaks: Hello @amosonn! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





* In the file [`pydicom/tests/test_dataset.py`](https://github.com/pydicom/pydicom/blob/1ae35df6e1c8b565c7838dd84c594fd22f66b768/pydicom/tests/test_dataset.py):

> [Line 1522:1](https://github.com/pydicom/pydicom/blob/1ae35df6e1c8b565c7838dd84c594fd22f66b768/pydicom/tests/test_dataset.py#L1522): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1


darcymason: See my comment in discussion above.  Also, @amosonn, please add a comment in the what's-new .rst for v1.3 - it will change outputs that people may see, so worth making sure it is flagged for awareness.

Patch:
diff --git a/appveyor.yml b/appveyor.yml
index a2976a1bb..7db9f5f31 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -33,7 +33,7 @@ install:
 
   # Installed prebuilt dependencies from conda
   - "conda install pip numpy nose wheel pytest matplotlib -y -q"
-  - "conda install -y -q setuptools"
+  - "conda --version"
   - "pip install nose-timer nose-exclude"
 
   # Install other pydicom dependencies
diff --git a/build_tools/travis/install.sh b/build_tools/travis/install.sh
index 1e2b19ea2..76850e53d 100755
--- a/build_tools/travis/install.sh
+++ b/build_tools/travis/install.sh
@@ -32,7 +32,8 @@ if [[ "$DISTRIB" == "conda" ]]; then
     MINICONDA_PATH=/home/travis/miniconda
     chmod +x miniconda.sh && ./miniconda.sh -b -p $MINICONDA_PATH
     export PATH=$MINICONDA_PATH/bin:$PATH
-    conda update --yes conda
+    conda --version
+    # conda update --yes conda
 
     # Configure the conda environment and put it in the path using the
     # provided versions
diff --git a/doc/base_element.rst b/doc/base_element.rst
index b8b4fb426..bdb5b6099 100644
--- a/doc/base_element.rst
+++ b/doc/base_element.rst
@@ -61,6 +61,13 @@ number::
 In the latter case (using the tag number directly) a DataElement instance is
 returned, so the ``.value`` must be used to get the value.
 
+.. note::
+
+    In pydicom, private data elements are displayed with square brackets
+    around the name (if the name is known to pydicom).  These are shown for convenience only;
+    the descriptive name in brackets cannot be used to retrieve data elements.
+    See details in :doc:`private_data_elements`.
+
 You can also set values by name (DICOM keyword) or tag number::
 
   >>> ds.PatientID = "12345"
@@ -95,22 +102,22 @@ convention::
 
   >>> ds.BeamSequence[0].BeamName
   'Field 1'
+  >>> # Or, set an intermediate variable to a dataset in the list
+  >>> beam1 = ds.BeamSequence[0]  # First dataset in the sequence
+  >>> beam1.BeamName
+  'Field 1'
 
 See :ref:`sphx_glr_auto_examples_metadata_processing_plot_sequences.py`.
 
 Using DICOM keywords is the recommended way to access data elements, but you
 can also use the tag numbers directly, such as::
 
-  >>> # Same thing with tag numbers:
+  >>> # Same thing with tag numbers - much harder to read:
+  >>> # Really should only be used if DICOM keyword not in pydicom dictionary
   >>> ds[0x300a,0xb0][0][0x300a,0xc2].value
   'Field 1'
-  >>> # yet another way, using another variable
-  >>> beam1=ds[0x300a,0xb0][0]
-  >>> beam1.BeamName, beam1[0x300a,0xc2].value
-  ('Field 1', 'Field 1')
 
-
-If you don't remember or know the exact tag name, :class:`dataset.Dataset`
+If you don't remember or know the exact tag name (aka DICOM keyword), :class:`dataset.Dataset`
 provides a handy :func:`dataset.Dataset.dir` method, useful during interactive
 sessions at the Python prompt::
 
@@ -126,8 +133,11 @@ have the specified string anywhere in the name (case insensitive).
    available in the dataset.
 
 You can also see all the names that pydicom knows about by viewing the
-``_dicom_dict.py`` file. You could modify that file to add tags that pydicom
-doesn't already know about.
+``_dicom_dict.py`` file. It should not normally be necessary, but you can add
+your own entries to the DICOM dictionary at run time using :func:`datadict.add_dict_entries`
+or :func:`datadict.add_dict_entry`.  Similarly, you can add private data elements
+to the private dictionary using :func:`datadict.add_private_dict_entries` or
+:func:`datadict.add_private_dict_entries`.
 
 Under the hood, :class:`dataset.Dataset` stores a DataElement object for each
 item, but when accessed by name (e.g. ``ds.PatientName``) only the ``value`` of
@@ -152,7 +162,7 @@ use the `in` keyword::
   >>> "PatientName" in ds
   True
 
-To remove a data element from the dataset,  use :func:`dataset.Dataset.del`::
+To remove a data element from the dataset,  use python's `del` statement::
 
   >>> del ds.SoftwareVersions   # or del ds[0x0018, 0x1020]
 
@@ -163,7 +173,7 @@ To work with pixel data, the raw bytes are available through the usual tag::
   >>> ds = pydicom.dcmread(filename)
   >>> pixel_bytes = ds.PixelData
 
-but to work with them in a more intelligent way, use ``pixel_array``
+but to work with them in a more intelligent way, use :func:`Dataset.pixel_array`
 (requires the `NumPy library <http://numpy.org>`_)::
 
   >>> pix = ds.pixel_array
@@ -197,21 +207,26 @@ which stores the following things:
 Tag
 ---
 
+Tag is not generally used directly in user code, as Tags are automatically created 
+when you assign or read data elements using the DICOM keywords as illustrated in
+sections above.
+
 The Tag class is derived from Python's ``int``, so in effect, it is just
 a number with some extra behaviour:
 
   * Tag enforces that the DICOM tag fits in the expected 4-byte (group,element)
-  * A Tag instance can be created from an int or from a tuple containing
-    the (group,element) separately::
+  * A Tag instance can be created from an int or a tuple containing
+    the (group,element), or from the DICOM keyword::
 
       >>> from pydicom.tag import Tag
-      >>> t1=Tag(0x00100010) # all of these are equivalent
-      >>> t2=Tag(0x10,0x10)
-      >>> t3=Tag((0x10, 0x10))
+      >>> t1 = Tag(0x00100010) # all of these are equivalent
+      >>> t2 = Tag(0x10,0x10)
+      >>> t3 = Tag((0x10, 0x10))
+      >>> t4 = Tag("PatientName")
       >>> t1
       (0010, 0010)
-      >>> t1==t2, t1==t3
-      (True, True)
+      >>> t1==t2, t1==t3, t1==t4
+      (True, True, True)
 
   * Tag has properties group and element (or elem) to return the group and
     element portions
@@ -224,3 +239,6 @@ Sequence
 Sequence is derived from Python's ``list``. The only added functionality is
 to make string representations prettier. Otherwise all the usual methods of
 ``list`` like item selection, append, etc. are available.
+
+For examples of accessing data nested in sequences, see
+:ref:`sphx_glr_auto_examples_metadata_processing_plot_sequences.py`.
diff --git a/doc/getting_started.rst b/doc/getting_started.rst
index 1a13961e9..b451657bb 100644
--- a/doc/getting_started.rst
+++ b/doc/getting_started.rst
@@ -77,7 +77,7 @@ Prerequisites
    * gdcm
    * jpeg_ls
    * jpeg2000
-   * pytest (if running pydicom's test suite)
+   * pytest (if running pydicom's test suite). pytest<5 if in Python 2.
 
 
 Installing pydicom
@@ -139,6 +139,8 @@ pydicom's setup.py file and::
 
 This will install `pytest <https://pytest.org>`_ if it is not 
 already installed.
+
+In v1.3 run under Python 2, if pytest is not found, please `python2 -m pip install "pytest<5"`
   
 Or, in linux you can also use::
 
diff --git a/doc/private_data_elements.rst b/doc/private_data_elements.rst
new file mode 100644
index 000000000..ce8dadd4e
--- /dev/null
+++ b/doc/private_data_elements.rst
@@ -0,0 +1,115 @@
+.. _private_data_elements:
+
+Private Data Elements
+=====================
+
+.. rubric:: Accessing or creating private data elements
+
+.. currentmodule:: pydicom.dataset
+
+Introduction
+------------
+
+The DICOM standard allows DICOM file creators to use `private data elements` to store
+information that is not defined by the DICOM standard itself.
+
+Private data elements are stored in Datasets just like other data elements.
+When reading files with pydicom, they will automatically be read
+and available for display.  Pydicom knows descriptive names for some 'well-known'
+private data elements, but for others it may not be able to show anything except
+the tag and the value.
+
+When writing your own private data elements, the DICOM standard requires the use
+of 'private creator blocks'.  Pydicom (since v1.3) has some convenience functions
+to make creating private blocks and data elements easier.
+
+The sections below outlines accessing and creating private blocks and data elements
+using pydicom.
+
+Displaying Private Data Elements in pydicom
+-------------------------------------------
+
+Here is an example of some private tags displayed for pydicom's test file 'CT_small.dcm'::
+
+    >>> from pydicom import dcmread
+    >>> from pydicom.data import get_testdata_files
+    >>> ct_filename = get_testdata_files("CT_small")[0]
+    >>> ds = dcmread(ct_filename)
+    >>> ds
+    (0008, 0005) Specific Character Set              CS: 'ISO_IR 100'
+    (0008, 0008) Image Type                          CS: ['ORIGINAL', 'PRIMARY', 'AX
+    IAL']
+    ...
+    ...
+    (0009, 0010) Private Creator                     LO: 'GEMS_IDEN_01'
+    (0009, 1001) [Full fidelity]                     LO: 'GE_GENESIS_FF'
+    (0009, 1002) [Suite id]                          SH: 'CT01'
+    ...
+
+The last two lines in the example above show pydicom's display of two private data
+elements.  The line preceding those shows the private creator data element
+that reserves a section of tag element numbers for that creator's use.
+
+Since the descriptions for private data elements are not part of the DICOM standard,
+and are thus not necessarily unique,
+pydicom does not allow you to access data elements using those names. This is indicated
+by enclosing the text in square brackets, to make it clear it is different from DICOM 
+standard descriptors.
+
+You can still access the private data elements using the tag, remembering that 
+data elements access by tag number return a full DataElement instance, and the `value` 
+attribute is needed to get the value::
+
+    >>> ds[0x00091001].value
+    'GE_GENESIS_FF'
+
+You can also create a :class:`PrivateBlock` instance and access elements through it::
+
+    >>> block = ds.private_block(0x0009, 'GEMS_IDEN_01')
+    >>> block[0x01]
+    (0009, 1001) [Full fidelity]                     LO: 'GE_GENESIS_FF'
+    >>> block[0x01].value
+    'GE_GENESIS_FF'
+
+Using the private block like this is even more useful when creating your own private data
+elements, as shown in the next section.
+
+Setting Private Data Elements with pydicom
+------------------------------------------
+
+The DICOM standard requires a private creator data element to identify and reserve a section
+of private tags. That name should be unique, and usually has the company name as the first part 
+to accomplish that.  Pydicom (since v1.3) provides convenient functions to manage this::
+
+    >>> block = ds.private_block(0x000b, "My company 001", create=True)
+    >>> block.add_new(0x01, "SH", "my value")
+    >>> ds
+    ...
+    (000b, 0010) Private Creator                     LO: 'My company 001'
+    (000b, 1001) Private tag data                    SH: 'my value'
+    ...
+
+Standard python operations like `in` and `del` can also be used when working 
+with block object::
+
+    >>> 0x01 in block
+    True
+    >>> 0x02 in block
+    False
+    >>> del block[0x01]
+    >>> 0x01 in block
+    False
+
+Removing All Private Data Elements
+-----------------------------------------------
+
+One part of anonymizing a DICOM file is to ensure that private data elements
+have been removed, as there is no guarantee as to what kind of information
+might be contained in them.  Pydicom provides a convenient function 
+:func:`Dataset.remove_private_tags` to recursively remove private elements::
+
+    >>> ds.remove_private_tags()
+
+This can also be helpful during interactive sessions when exploring DICOM files,
+to remove a large number of lines from the display of a dataset --
+lines which may not provide useful information.
diff --git a/doc/pydicom_user_guide.rst b/doc/pydicom_user_guide.rst
index b9e07a803..d252b0724 100644
--- a/doc/pydicom_user_guide.rst
+++ b/doc/pydicom_user_guide.rst
@@ -6,9 +6,12 @@ Pydicom User Guide
 
 .. toctree::
 
-   transition_to_pydicom1.rst
+   warn_transition_to_pydicom1.rst
    base_element.rst
    writing_files.rst
    working_with_pixel_data.rst
    image_data_handlers.rst
-   viewing_images.rst
\ No newline at end of file
+   viewing_images.rst
+   private_data_elements.rst
+   transition_to_pydicom1.rst
+   python2_support.rst
\ No newline at end of file
diff --git a/doc/python2_support.rst b/doc/python2_support.rst
new file mode 100644
index 000000000..5c2c14a3a
--- /dev/null
+++ b/doc/python2_support.rst
@@ -0,0 +1,42 @@
+.. _Python2_support:
+
+Python 2 Support Plan for Pydicom
+=================================
+
+.. rubric:: Timeline of support for Python 2
+
+The Python developers have stated that Python 2 will no longer be supported starting Jan 1, 2020.
+Numpy is also dropping support for Python 2 at that time. Pytest, which pydicom uses for testing,
+will support only Python 3.5+
+starting with the pytest 5.0 release, but are supporting the 4.6 version until mid 2020.
+
+It is clear -- Python 2 will become a thing of the past starting Jan 2020.
+All packages, including pydicom, need to think of transitioning away from Python 2 and
+supporting only Python 3.
+
+Pydicom code was written with common code for both Python 2.7 and Python 3 as much as possible.
+Where necessary, checks for Python 2 have been used to create small blocks of distinct
+code.  When the time comes, it should be relatively easy to remove the Python 2 blocks and leave
+a Python-3-only version of pydicom.
+
+After some discussion on github, the proposed plan for pydicom and Python 2 support is as follows
+
+* pydicom v1.3 (July 2019) - no changes to Python versions supported. Adds a deprecation
+  warning (to be deprecated in v1.5) when run under Python 2
+
+* pydicom v1.4 (planned for release in December 2019) will support Python 2.7,
+  with deprecation warning as above.
+
+* pydicom v1.5 (planned for ~April 2020) officially will not support Python 2.7,
+  but no compatibility code will be removed from pydicom; it will include a stronger 
+  'deprecated' (past tense) message.  It will not install using pip, but those who
+  choose to do so could download and install themselves; however, we make no guarantees
+  that it will function correctly.
+
+* pydicom v1.6 (no date set) will remove Python 2 code from pydicom, and officially
+  drop Python 3.4 support as well. Any new pydicom code afterwards can target
+  Python 3.5+ features. 
+
+We may consider the possibility of backporting some fixes to pydicom v1.4 for very serious issues,
+if users make the case for it.  Generally speaking, however, we encourage all pydicom users to make
+the transition to Python 3 by early 2020.
diff --git a/doc/ref_guide.rst b/doc/ref_guide.rst
index 02fee3f6e..c58dfa0ac 100644
--- a/doc/ref_guide.rst
+++ b/doc/ref_guide.rst
@@ -27,10 +27,10 @@ File Writing
 ============
 
 DICOM files can also be written using pydicom. There are two ways to do this.
-The first is to use ``write_file`` with a prexisting FileDataset (derived from Dataset) instance.
+The first is to use ``dcmwrite`` with a prexisting FileDataset (derived from Dataset) instance.
 The second is to use the ``save_as`` method on an Dataset instance.
 
-.. autofunction:: pydicom.filewriter.write_file
+.. autofunction:: pydicom.filewriter.dcmwrite
 
 .. automethod:: pydicom.dataset.Dataset.save_as
 
diff --git a/doc/warn_transition_to_pydicom1.rst b/doc/warn_transition_to_pydicom1.rst
new file mode 100644
index 000000000..2030b5604
--- /dev/null
+++ b/doc/warn_transition_to_pydicom1.rst
@@ -0,0 +1,12 @@
+For users of pydicom < 1.0
+========================
+
+.. rubric:: Warning about pydicom < 1.0 and reference to Transition to Pydicom 1 section
+
+Pydicom < 1
+-------
+
+.. warning::
+
+        Pydicom versions 1.0 and higher introduce backwards-incompatible changes.
+        If you have used earlier versions of pydicom, see :doc:`transition_to_pydicom1` for details.
diff --git a/doc/whatsnew/v1.3.0.rst b/doc/whatsnew/v1.3.0.rst
index 18d2b33dd..5b06b5147 100644
--- a/doc/whatsnew/v1.3.0.rst
+++ b/doc/whatsnew/v1.3.0.rst
@@ -26,9 +26,15 @@ Enhancements
 * Added possibility to get and add private tags without adding them to the
   private dictionary
 * Added possibility to use a ``Dataset`` in a ``NumPy`` array
+* Allow missing padding byte in Pixel Data, issue a warning in this case
+  (:issue:`864`)
 
 Fixes
 .....
+* Correctly handle `Dataset.pop` and `Dataset.setdefault` for tuple and
+  keyword arguments (:issue:`852`)
+* Correctly handle encoding errors when any of the encodings are invalid
+  (not just the first) (:issue:`850`)
 * Do not raise while resolving an ambiguous VR dependent on
   `PixelRepresentation` if both `PixelRepresentation` and `PixelData` are
   not present (:issue:`838`)
diff --git a/pydicom/__init__.py b/pydicom/__init__.py
index de10167e1..e00e000d7 100644
--- a/pydicom/__init__.py
+++ b/pydicom/__init__.py
@@ -47,3 +47,9 @@ __all__ = ['DataElement',
            'write_file',
            '__version__',
            '__version_info__']
+
+from pydicom.compat import in_py2
+if in_py2:
+    import warnings
+    msg = 'Python 2 will no longer be supported after the pydicom v1.4 release'
+    warnings.warn(msg, DeprecationWarning)
diff --git a/pydicom/charset.py b/pydicom/charset.py
index 9d11d3f24..676730c22 100644
--- a/pydicom/charset.py
+++ b/pydicom/charset.py
@@ -1,11 +1,12 @@
 # Copyright 2008-2018 pydicom authors. See LICENSE file for details.
 """Handle alternate character sets for character strings."""
+import codecs
 import re
 import warnings
 
 from pydicom import compat, config
-from pydicom.valuerep import PersonNameUnicode, text_VRs, TEXT_VR_DELIMS
 from pydicom.compat import in_py2
+from pydicom.valuerep import PersonNameUnicode, text_VRs, TEXT_VR_DELIMS
 
 # default encoding if no encoding defined - corresponds to ISO IR 6 / ASCII
 default_encoding = "iso8859"
@@ -85,6 +86,7 @@ CODES_TO_ENCODINGS = {
 }
 
 ENCODINGS_TO_CODES = {v: k for k, v in CODES_TO_ENCODINGS.items()}
+ENCODINGS_TO_CODES['shift_jis'] = ESC + b')I'
 
 # Multi-byte character sets except Korean are handled by Python.
 # To decode them, the escape sequence shall be preserved in the input byte
@@ -94,6 +96,182 @@ handled_encodings = ('iso2022_jp',
                      'iso_ir_58')
 
 
+def _encode_to_jis_x_0201(value, errors='strict'):
+    """Convert a unicode string into JIS X 0201 byte string using shift_jis
+    encodings.
+    shift_jis is a superset of jis_x_0201. So we can regard the encoded value
+    as jis_x_0201 if it is single byte character.
+
+    Parameters
+    ----------
+    value : text type
+        The unicode string as presented to the user.
+    errors : str
+        The behavior of a character which could not be encoded. If 'strict' is
+        passed, raise an UnicodeEncodeError. If any other value is passed,
+        non ISO IR 14 characters are replaced by the ASCII '?'.
+
+    Returns
+    -------
+    byte string
+        The encoded string. If some characters in value could not be encoded to
+        JIS X 0201, and `errors` is not set to 'strict', they are replaced to
+        '?'.
+
+    Raises
+    ------
+    UnicodeEncodeError
+        If errors is set to 'strict' and `value` could not be encoded with
+        JIS X 0201.
+    """
+
+    Encoder = codecs.getincrementalencoder('shift_jis')
+    encoder = Encoder()
+
+    # If errors is not strict, this function is used as fallback.
+    # In this case, we use only ISO IR 14 to encode given value
+    # without escape sequence.
+    if errors != 'strict' or value == '':
+        encoded = b''
+        for c in value:
+            try:
+                b = encoder.encode(c)
+            except UnicodeEncodeError as e:
+                b = b'?'
+
+            if len(b) != 1 or 0x80 <= ord(b):
+                b = b'?'
+            encoded += b
+        return encoded
+
+    encoded = encoder.encode(value[0])
+    if len(encoded) != 1:
+        raise UnicodeEncodeError(
+            'shift_jis', value, 0, len(value), 'illegal multibyte sequence')
+
+    msb = ord(encoded) & 0x80  # msb is 1 for ISO IR 13, 0 for ISO IR 14
+    for i, c in enumerate(value[1:], 1):
+        try:
+            b = encoder.encode(c)
+        except UnicodeEncodeError as e:
+            e.start = i
+            e.end = len(value)
+            raise e
+        if len(b) != 1 or ((ord(b) & 0x80) ^ msb) != 0:
+            character_set = 'ISO IR 14' if msb == 0 else 'ISO IR 13'
+            msg = 'Given character is out of {}'.format(character_set)
+            raise UnicodeEncodeError('shift_jis', value, i, len(value), msg)
+        encoded += b
+
+    return encoded
+
+
+def _encode_to_jis_x_0208(value, errors='strict'):
+    """Convert a unicode string into JIS X 0208 byte string using iso2022_jp
+    encodings.
+    The escape sequence which is located at the end of the encoded value has
+    to vary depending on the value 1 of SpecificCharacterSet. So we have to
+    trim it and append the correct escape sequence manually.
+
+    Parameters
+    ----------
+    value : text type
+        The unicode string as presented to the user.
+    errors : str
+        The behavior of a character which could not be encoded. This value
+        is passed to errors argument of str.encode().
+
+    Returns
+    -------
+    byte string
+        The encoded string. If some characters in value could not be encoded to
+        JIS X 0208, it depends on the behavior of iso2022_jp encoder.
+
+    Raises
+    ------
+    UnicodeEncodeError
+        If errors is set to 'strict' and `value` could not be encoded with
+        JIS X 0208.
+    """
+
+    # If errors is not strict, this function is used as fallback.
+    # So keep the tail escape sequence of encoded for backward compatibility.
+    if errors != 'strict':
+        return value.encode('iso2022_jp', errors=errors)
+
+    Encoder = codecs.getincrementalencoder('iso2022-jp')
+    encoder = Encoder()
+
+    encoded = encoder.encode(value[0])
+    if encoded[:3] != ENCODINGS_TO_CODES['iso2022_jp']:
+        raise UnicodeEncodeError(
+            'iso2022_jp', value, 0, len(value),
+            'Given character is out of ISO IR 87')
+
+    for i, c in enumerate(value[1:], 1):
+        try:
+            b = encoder.encode(c)
+        except UnicodeEncodeError as e:
+            e.start = i
+            e.end = len(value)
+            raise e
+        if b[:3] == ENCODINGS_TO_CODES['iso8859']:
+            raise UnicodeEncodeError(
+                'iso2022_jp', value, i, len(value),
+                'Given character is out of ISO IR 87')
+        encoded += b
+    return encoded
+
+
+def _get_escape_sequence_for_encoding(encoding, encoded=None):
+    """ Return an escape sequence corresponding to the given encoding. If
+    encoding is 'shift_jis', return 'ESC)I' or 'ESC(J' depending on the first
+    byte of encoded.
+
+    Parameters
+    ----------
+    encoding : str
+        An encoding is used to specify  an escape sequence.
+
+    encoded : bytes or str
+        The encoded value is used to chose an escape sequence if encoding is
+        'shift_jis'
+
+    Returns
+    -------
+    string
+        Escape sequence for encoded value.
+    """
+
+    ESC_ISO_IR_14 = ESC + b'(J'
+    ESC_ISO_IR_13 = ESC + b')I'
+
+    if encoding == 'shift_jis':
+        if encoded is None:
+            return ESC_ISO_IR_14
+
+        if not in_py2:
+            first_byte = encoded[0]
+        else:
+            first_byte = ord(encoded[0])
+        if 0x80 <= first_byte:
+            return ESC_ISO_IR_13
+
+        return ESC_ISO_IR_14
+    return ENCODINGS_TO_CODES.get(encoding, b'')
+
+
+# These encodings need escape sequence to handle alphanumeric characters.
+need_tail_escape_sequence_encodings = ('iso2022_jp', 'iso-2022-jp')
+
+
+custom_encoders = {
+    'shift_jis': _encode_to_jis_x_0201,
+    'iso2022_jp': _encode_to_jis_x_0208,
+    'iso-2022-jp': _encode_to_jis_x_0208
+}
+
+
 def decode_string(value, encodings, delimiters):
     """Convert a raw byte string into a unicode string using the given
     list of encodings.
@@ -280,9 +458,14 @@ def encode_string(value, encodings):
     """
     for i, encoding in enumerate(encodings):
         try:
-            encoded = value.encode(encoding)
+            encoded = _encode_string_impl(value, encoding)
+
             if i > 0 and encoding not in handled_encodings:
-                return ENCODINGS_TO_CODES.get(encoding, b'') + encoded
+                escape_sequence = _get_escape_sequence_for_encoding(
+                        encoding, encoded=encoded)
+                encoded = escape_sequence + encoded
+            if encoding in need_tail_escape_sequence_encodings:
+                encoded += _get_escape_sequence_for_encoding(encodings[0])
             return encoded
         except UnicodeError:
             continue
@@ -303,7 +486,7 @@ def encode_string(value, encodings):
         warnings.warn("Failed to encode value with encodings: {} - using "
                       "replacement characters in encoded string"
                       .format(', '.join(encodings)))
-        return value.encode(encodings[0], errors='replace')
+        return _encode_string_impl(value, encodings[0], errors='replace')
 
 
 def _encode_string_parts(value, encodings):
@@ -336,14 +519,14 @@ def _encode_string_parts(value, encodings):
     """
     encoded = bytearray()
     unencoded_part = value
+    best_encoding = None
     while unencoded_part:
         # find the encoding that can encode the longest part of the rest
         # of the string still to be encoded
         max_index = 0
-        best_encoding = None
         for encoding in encodings:
             try:
-                unencoded_part.encode(encoding)
+                _encode_string_impl(unencoded_part, encoding)
                 # if we get here, the whole rest of the value can be encoded
                 best_encoding = encoding
                 max_index = len(unencoded_part)
@@ -354,20 +537,36 @@ def _encode_string_parts(value, encodings):
                     max_index = e.start
                     best_encoding = encoding
         # none of the given encodings can encode the first character - give up
-        if best_encoding is None:
-            raise ValueError()
+        if max_index == 0:
+            raise ValueError("None of the given encodings can encode the "
+                             "first character")
 
         # encode the part that can be encoded with the found encoding
-        encoded_part = unencoded_part[:max_index].encode(best_encoding)
+        encoded_part = _encode_string_impl(unencoded_part[:max_index],
+                                           best_encoding)
         if best_encoding not in handled_encodings:
-            encoded += ENCODINGS_TO_CODES.get(best_encoding, b'')
+            encoded += _get_escape_sequence_for_encoding(
+                    best_encoding, encoded=encoded_part)
         encoded += encoded_part
         # set remaining unencoded part of the string and handle that
         unencoded_part = unencoded_part[max_index:]
     # unencoded_part is empty - we are done, return the encoded string
+    if best_encoding in need_tail_escape_sequence_encodings:
+        encoded += _get_escape_sequence_for_encoding(encodings[0])
     return encoded
 
 
+def _encode_string_impl(value, encoding, errors='strict'):
+    """Convert a unicode string into a byte string. If given encoding is in
+    custom_encoders, use a corresponding custom_encoder. If given encoding
+    is not in custom_encoders, use a corresponding python handled encoder.
+    """
+    if encoding in custom_encoders:
+        return custom_encoders[encoding](value, errors=errors)
+    else:
+        return value.encode(encoding, errors=errors)
+
+
 # DICOM PS3.5-2008 6.1.1 (p 18) says:
 #   default is ISO-IR 6 G0, equiv to common chr set of ISO 8859 (PS3.5 6.1.2.1)
 #    (0008,0005)  value 1 can *replace* the default encoding...
@@ -384,9 +583,12 @@ def _encode_string_parts(value, encodings):
 def convert_encodings(encodings):
     """Converts DICOM encodings into corresponding python encodings.
     Handles some common spelling mistakes and issues a warning in this case.
-    Handled stand-alone encodings: if they are the first encodings,
+    Handles stand-alone encodings: if they are the first encodings,
     additional encodings are ignored, if they are not the first encoding,
     they are ignored. In both cases, a warning is issued.
+    Invalid encodings are replaced with the default encoding with a
+    respective warning issued, if `config.enforce_valid_values` is `False`,
+    otherwise an exception is raised.
 
     Parameters
     ----------
@@ -397,8 +599,16 @@ def convert_encodings(encodings):
     -------
     list of str
         The list of Python encodings corresponding to the DICOM encodings.
-        If conversion fails, `encodings` is returned unchanged, assuming
-        that it already has been converted to Python encodings.
+        If an encoding is already a Python encoding, it is returned unchanged.
+        Encodings with common spelling errors are replaced by the correct
+        encoding, and invalid encodings are replaced with the default
+        encoding if `config.enforce_valid_values` is `False`.
+
+    Raises
+    ------
+    LookupError
+        In case of an invalid encoding that could not be corrected if
+        `config.enforce_valid_values` is set.
     """
 
     # If a list if passed, we don't want to modify the list in place so copy it
@@ -409,58 +619,96 @@ def convert_encodings(encodings):
     elif not encodings[0]:
         encodings[0] = 'ISO_IR 6'
 
-    try:
-        py_encodings = [python_encoding[x] for x in encodings]
-
-    except KeyError:
-        # check for some common mistakes in encodings
-        patched_encodings = []
-        patched = {}
-        for x in encodings:
-            # check for spelling errors, but exclude the correct spelling
-            # standard encodings
-            if re.match('^ISO[^_]IR', x) is not None:
-                patched[x] = 'ISO_IR' + x[6:]
-                patched_encodings.append(patched[x])
-            # encodings with code extensions
-            elif re.match('^(?=ISO.2022.IR.)(?!ISO 2022 IR )', x) is not None:
-                patched[x] = 'ISO 2022 IR ' + x[12:]
-                patched_encodings.append(patched[x])
-            else:
-                patched_encodings.append(x)
-        # fallback: assume that it is already a python encoding
-        py_encodings = encodings
-        if patched:
-            try:
-                encodings = patched_encodings
-                py_encodings = [python_encoding[x] for x in encodings]
-                for old, new in patched.items():
-                    warnings.warn("Incorrect value for Specific Character Set "
-                                  "'{}' - assuming '{}'".format(old, new),
-                                  stacklevel=2)
-            except KeyError:
-                pass
-        # if patching failed at this point, the original encodings
-        # will be returned, assuming that they are already Python encodings;
-        # otherwise, a LookupError will be raised in the using code
+    py_encodings = []
+    for encoding in encodings:
+        try:
+            py_encodings.append(python_encoding[encoding])
+        except KeyError:
+            py_encodings.append(
+                _python_encoding_for_corrected_encoding(encoding))
 
-    # handle illegal stand-alone encodings
     if len(encodings) > 1:
-        if encodings[0] in STAND_ALONE_ENCODINGS:
-            warnings.warn("Value '{}' for Specific Character Set does not "
-                          "allow code extensions, ignoring: {}"
-                          .format(encodings[0], ', '.join(encodings[1:])),
-                          stacklevel=2)
-            py_encodings = py_encodings[:1]
-        else:
-            for i, encoding in reversed(list(enumerate(encodings[1:]))):
-                if encoding in STAND_ALONE_ENCODINGS:
-                    warnings.warn(
-                        "Value '{}' cannot be used as code extension, "
-                        "ignoring it".format(encoding),
-                        stacklevel=2)
-                    del py_encodings[i + 1]
+        py_encodings = _handle_illegal_standalone_encodings(encodings,
+                                                            py_encodings)
+    return py_encodings
 
+
+def _python_encoding_for_corrected_encoding(encoding):
+    """Try to replace the given invalid encoding with a valid encoding by
+    checking for common spelling errors, and return the correct Python
+    encoding for that encoding. Otherwise check if the
+    encoding is already a valid Python encoding, and return that. If both
+    attempts fail, return the default encoding.
+    Issue a warning for the invalid encoding except for the case where it is
+    already converted.
+    """
+    # standard encodings
+    patched = None
+    if re.match('^ISO[^_]IR', encoding) is not None:
+        patched = 'ISO_IR' + encoding[6:]
+    # encodings with code extensions
+    elif re.match('^(?=ISO.2022.IR.)(?!ISO 2022 IR )',
+                  encoding) is not None:
+        patched = 'ISO 2022 IR ' + encoding[12:]
+
+    if patched:
+        # handle encoding patched for common spelling errors
+        try:
+            py_encoding = python_encoding[patched]
+            _warn_about_invalid_encoding(encoding, patched)
+            return py_encoding
+        except KeyError:
+            _warn_about_invalid_encoding(encoding)
+            return default_encoding
+
+    # fallback: assume that it is already a python encoding
+    try:
+        codecs.lookup(encoding)
+        return encoding
+    except LookupError:
+        _warn_about_invalid_encoding(encoding)
+        return default_encoding
+
+
+def _warn_about_invalid_encoding(encoding, patched_encoding=None):
+    """Issue a warning for the given invalid encoding.
+    If patched_encoding is given, it is mentioned as the
+    replacement encoding, other the default encoding.
+    If no replacement encoding is given, and config.enforce_valid_values
+    is set, LookupError is raised.
+    """
+    if patched_encoding is None:
+        if config.enforce_valid_values:
+            raise LookupError(
+                "Unknown encoding '{}'".format(encoding))
+        msg = ("Unknown encoding '{}' - using default encoding "
+               "instead".format(encoding))
+    else:
+        msg = ("Incorrect value for Specific Character Set "
+               "'{}' - assuming '{}'".format(encoding, patched_encoding))
+    warnings.warn(msg, stacklevel=2)
+
+
+def _handle_illegal_standalone_encodings(encodings, py_encodings):
+    """Check for stand-alone encodings in multi-valued encodings.
+    If the first encoding is a stand-alone encoding, the rest of the
+    encodings is removed. If any other encoding is a stand-alone encoding,
+    it is removed from the encodings.
+    """
+    if encodings[0] in STAND_ALONE_ENCODINGS:
+        warnings.warn("Value '{}' for Specific Character Set does not "
+                      "allow code extensions, ignoring: {}"
+                      .format(encodings[0], ', '.join(encodings[1:])),
+                      stacklevel=2)
+        py_encodings = py_encodings[:1]
+    else:
+        for i, encoding in reversed(list(enumerate(encodings[1:]))):
+            if encoding in STAND_ALONE_ENCODINGS:
+                warnings.warn(
+                    "Value '{}' cannot be used as code extension, "
+                    "ignoring it".format(encoding),
+                    stacklevel=2)
+                del py_encodings[i + 1]
     return py_encodings
 
 
diff --git a/pydicom/dataelem.py b/pydicom/dataelem.py
index d9c1609e9..ad69b3a0b 100644
--- a/pydicom/dataelem.py
+++ b/pydicom/dataelem.py
@@ -302,10 +302,17 @@ class DataElement(object):
     @property
     def repval(self):
         """Return a str representation of the element's `value`."""
-        byte_VRs = ['OB', 'OW', 'OW/OB', 'OW or OB', 'OB or OW',
-                    'US or SS or OW', 'US or SS']
-        if (self.VR in byte_VRs and len(self.value) > self.maxBytesToDisplay):
-            repVal = "Array of %d bytes" % len(self.value)
+        long_VRs = {"OB", "OD", "OF", "OW", "UN", "UT"}
+        if set(self.VR.split(" or ")) & long_VRs:
+            try:
+                length = len(self.value)
+            except TypeError:
+                pass
+            else:
+                if length > self.maxBytesToDisplay:
+                    return "Array of %d elements" % length
+        if self.VM > self.maxBytesToDisplay:
+            repVal = "Array of %d elements" % self.VM
         elif isinstance(self.value, UID):
             repVal = self.value.name
         else:
diff --git a/pydicom/dataset.py b/pydicom/dataset.py
index b91ab5cc3..c30ada8db 100644
--- a/pydicom/dataset.py
+++ b/pydicom/dataset.py
@@ -25,7 +25,7 @@ from itertools import takewhile
 import pydicom  # for dcmwrite
 import pydicom.charset
 import pydicom.config
-from pydicom import compat
+from pydicom import compat, datadict
 from pydicom._version import __version_info__
 from pydicom.charset import default_encoding, convert_encodings
 from pydicom.config import logger
@@ -447,18 +447,14 @@ class Dataset(dict):
         bool
             True if the DataElement is in the Dataset, False otherwise.
         """
-        if isinstance(name, (str, compat.text_type)):
-            tag = tag_for_keyword(name)
-        else:
-            try:
-                tag = Tag(name)
-            except Exception:
-                return False
+        try:
+            tag = Tag(name)
+        except (ValueError, OverflowError):
+            return False
         # Test against None as (0000,0000) is a possible tag
         if tag is not None:
             return tag in self._dict
-        else:
-            return name in self._dict  # will no doubt raise an exception
+        return name in self._dict  # will no doubt raise an exception
 
     def decode(self):
         """Apply character set decoding to all DataElements in the Dataset.
@@ -1089,14 +1085,98 @@ class Dataset(dict):
         """Delete all data elements."""
         self._dict.clear()
 
-    def pop(self, *args, **kwargs):
-        return self._dict.pop(*args, **kwargs)
+    def pop(self, key, *args):
+        """Emulate dictionary `pop`, but additionally support tag ID tuple
+        and DICOM keyword.
+
+        Removes the data element for `key` if it exists and returns it,
+        otherwise returns a default value if given or raises `KeyError`.
+
+        Parameters
+        ----------
+        key: int or str or 2-tuple
+            if tuple - the group and element number of the DICOM tag
+            if int - the combined group/element number
+            if str - the DICOM keyword of the tag
+
+        *args: zero or one argument
+            defines the behavior if no tag exists for `key`: if given,
+            it defines the return value, if not given, `KeyError` is raised
+
+        Returns
+        -------
+        The data element for `key` if it exists, or the default value if given.
+
+        Raises
+        ------
+        KeyError
+            If the key is not a valid tag ID or keyword.
+            If the tag does not exist and no default is given.
+        """
+        try:
+            tag = Tag(key)
+        except (ValueError, OverflowError):
+            return self._dict.pop(key, *args)
+        return self._dict.pop(tag, *args)
 
     def popitem(self):
         return self._dict.popitem()
 
-    def setdefault(self,  *args, **kwargs):
-        return self._dict.setdefault(*args, **kwargs)
+    def setdefault(self, key, default=None):
+        """Emulate dictionary `setdefault`, but additionally support
+        tag ID tuple and DICOM keyword for `key`, and data element value
+        for `default`.
+
+        .. usage:
+
+        >>> ds = Dataset()
+        >>> pname = ds.setdefault((0x0010, 0x0010), "Test")
+        >>> pname
+        (0010, 0010) Patient's Name                      PN: 'Test'
+        >>> pname.value
+        'Test'
+        >>> psex = ds.setdefault('PatientSex',
+        ...     DataElement(0x00100040, 'CS', 'F'))
+        >>> psex.value
+        'F'
+
+        Parameters
+        ----------
+        key: int or str or 2-tuple
+            if tuple - the group and element number of the DICOM tag
+            if int - the combined group/element number
+            if str - the DICOM keyword of the tag
+
+        default: DataElement or value type or None
+            The default value that is inserted and returned if no data
+            element exists for the given key.
+            If it is not of type DataElement, a DataElement is constructed
+            instead for the given tag ID and default as value. This is only
+            possible for known tags (e.g. tags found via the dictionary
+            lookup).
+
+        Returns
+        -------
+        The data element for `key` if it exists, or the default value if
+        it is a DataElement or None, or a DataElement constructed with
+        `default` as value.
+
+        Raises
+        ------
+        KeyError
+            If the key is not a valid tag ID or keyword.
+            If no tag exists for `key`, default is not a DataElement
+            and not None, and key is not a known DICOM tag.
+        """
+        if key in self:
+            return self[key]
+        if default is not None:
+            if not isinstance(default, DataElement):
+                tag = Tag(key)
+                vr = datadict.dictionary_VR(tag)
+                default = DataElement(Tag(key), vr, default)
+            self[key] = default
+        return default
 
     def convert_pixel_data(self):
         """Convert the Pixel Data to a numpy array internally.
diff --git a/pydicom/filereader.py b/pydicom/filereader.py
index 7bada88fc..12e1eec4c 100644
--- a/pydicom/filereader.py
+++ b/pydicom/filereader.py
@@ -31,92 +31,6 @@ from pydicom.util.hexutil import bytes2hex
 from pydicom.valuerep import extra_length_VRs
 
 
-class DicomIter(object):
-    """Iterator over DICOM data elements created from a file-like object"""
-
-    def __init__(self, fp, stop_when=None, force=False):
-        """Read the preamble and meta info and prepare
-           iterator for remainder of file.
-
-        Parameters
-        ----------
-        fp : an open DicomFileLike object, at start of file
-        force : boolean
-            Force reading of data. See ``read_file`` for
-             more parameter info.
-
-        Adds flags to fp: Big/Little-endian & Implicit/Explicit VR
-        """
-        self.fp = fp
-        self.stop_when = stop_when
-        self.preamble = preamble = read_preamble(fp, force)
-        self.has_header = has_header = (preamble is not None)
-        self.file_meta_info = Dataset()
-
-        if has_header:
-            self.file_meta_info = file_meta_info = _read_file_meta_info(fp)
-            transfer_syntax = file_meta_info.TransferSyntaxUID
-
-            if transfer_syntax == pydicom.uid.ExplicitVRLittleEndian:
-                self._is_implicit_VR = False
-                self._is_little_endian = True
-
-            elif transfer_syntax == pydicom.uid.ImplicitVRLittleEndian:
-                self._is_implicit_VR = True
-                self._is_little_endian = True
-
-            elif transfer_syntax == pydicom.uid.ExplicitVRBigEndian:
-                self._is_implicit_VR = False
-                self._is_little_endian = False
-
-            elif transfer_syntax == pydicom.uid.DeflatedExplicitVRLittleEndian:
-                # See PS3.6-2008 A.5 (p 71) -- when written, the entire dataset
-                # following the file metadata was prepared the normal way,
-                # then "deflate" compression applied.
-                # All that is needed here is to decompress and then
-                # use as normal in a file-like object
-                zipped = fp.read()
-
-                # -MAX_WBITS part is from comp.lang.python answer:
-                # groups.google.com/group/comp.lang.python/msg/e95b3b38a71e6799
-                unzipped = zlib.decompress(zipped, -zlib.MAX_WBITS)
-
-                # a file-like object
-                fp = BytesIO(unzipped)
-
-                # point to new object
-                self.fp = fp
-                self._is_implicit_VR = False
-                self._is_little_endian = True
-            else:
-                # Any other syntax should be Explicit VR Little Endian,
-                # e.g. all Encapsulated (JPEG etc) are ExplVR-LE
-                # by Standard PS 3.5-2008 A.4 (p63)
-                self._is_implicit_VR = False
-                self._is_little_endian = True
-
-        else:  # no header -- make assumptions
-            fp.TransferSyntaxUID = pydicom.uid.ImplicitVRLittleEndian
-            self._is_little_endian = True
-            self._is_implicit_VR = True
-
-        impl_expl = ("Explicit", "Implicit")[self._is_implicit_VR]
-        big_little = ("Big", "Little")[self._is_little_endian]
-        logger.debug("Using {0:s} VR, {1:s} Endian transfer syntax".format(
-                     impl_expl, big_little))
-
-    def __iter__(self):
-        tags = sorted(self.file_meta_info.keys())
-        for tag in tags:
-            yield self.file_meta_info[tag]
-
-        for data_element in data_element_generator(self.fp,
-                                                   self._is_implicit_VR,
-                                                   self._is_little_endian,
-                                                   stop_when=self.stop_when):
-            yield data_element
-
-
 def data_element_generator(fp,
                            is_implicit_VR,
                            is_little_endian,
diff --git a/pydicom/pixel_data_handlers/numpy_handler.py b/pydicom/pixel_data_handlers/numpy_handler.py
index ae3cd4a3d..46fc4446e 100644
--- a/pydicom/pixel_data_handlers/numpy_handler.py
+++ b/pydicom/pixel_data_handlers/numpy_handler.py
@@ -242,14 +242,20 @@ def get_pixeldata(ds, read_only=False):
     # Check that the actual length of the pixel data is as expected
     actual_length = len(ds.PixelData)
     # Correct for the trailing NULL byte padding for odd length data
+
     padded_expected_len = expected_len + expected_len % 2
     if actual_length < padded_expected_len:
-        raise ValueError(
-            "The length of the pixel data in the dataset doesn't match the "
-            "expected amount ({0} vs. {1} bytes). The dataset may be "
-            "corrupted or there may be an issue with the pixel data handler."
-            .format(actual_length, padded_expected_len)
-        )
+        if actual_length == expected_len:
+            warnings.warn(
+                "The pixel data length is odd and misses a padding byte.")
+        else:
+            raise ValueError(
+                "The length of the pixel data in the dataset ({} bytes) "
+                "doesn't match the expected length ({} bytes). "
+                "The dataset may be corrupted or there may be an issue "
+                "with the pixel data handler."
+                .format(actual_length, padded_expected_len)
+            )
     elif actual_length > padded_expected_len:
         # PS 3.5, Section 8.1.1
         msg = (
diff --git a/setup.py b/setup.py
index a6dcc0b4a..c6f151ff7 100755
--- a/setup.py
+++ b/setup.py
@@ -22,6 +22,9 @@ description = "Pure python package for DICOM medical file reading and writing"
 
 needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
 pytest_runner = ['pytest-runner'] if needs_pytest else []
+
+# in_py2 check in next line - pytest>=5 requires Python 3
+TESTS_REQUIRE = ['pytest<5'] if sys.version_info[0] == 2 else ['pytest']
 _py_modules = []
 if not have_dicom:
     _py_modules = ['dicom']
@@ -56,7 +59,6 @@ LICENSE = "MIT"
 VERSION = __version__
 REQUIRES = []
 SETUP_REQUIRES = pytest_runner
-TESTS_REQUIRE = ['pytest']
 
 # get long description from README.md
 BASE_PATH = os.path.abspath(os.path.dirname(__file__))


Test patch:
diff --git a/pydicom/tests/test_charset.py b/pydicom/tests/test_charset.py
index 2598e6018..007ab8e53 100644
--- a/pydicom/tests/test_charset.py
+++ b/pydicom/tests/test_charset.py
@@ -117,7 +117,8 @@ class TestCharset(object):
         ds = dcmread(get_testdata_files("CT_small.dcm")[0])
         ds.read_encoding = None
         ds.SpecificCharacterSet = 'Unsupported'
-        with pytest.raises(LookupError, match='unknown encoding: Unsupported'):
+        with pytest.raises(LookupError,
+                           match=u"Unknown encoding 'Unsupported'"):
             ds.decode()
 
     def test_decoding_with_specific_tags(self):
@@ -188,9 +189,6 @@ class TestCharset(object):
         encodings = ['iso_ir_126', 'iso_ir_144']
         assert encodings == pydicom.charset.convert_encodings(encodings)
 
-        encodings = ['ISO IR 199', 'ISO_IR 100']
-        assert encodings == pydicom.charset.convert_encodings(encodings)
-
     def test_bad_decoded_multi_byte_encoding(self):
         """Test handling bad encoding for single encoding"""
         elem = DataElement(0x00100010, 'PN',
@@ -397,3 +395,45 @@ class TestCharset(object):
         ds_out = dcmread(fp)
         # we expect UTF-8 encoding here
         assert b'Buc^J\xc3\xa9r\xc3\xb4me' == ds_out.get_item(0x00100010).value
+
+    def test_invalid_second_encoding(self):
+        # regression test for #850
+        elem = DataElement(0x00100010, 'PN', 'CITIZEN')
+        with pytest.warns(UserWarning,
+                          match=u"Unknown encoding 'ISO 2022 IR 146' "
+                                u"- using default encoding instead"):
+            pydicom.charset.decode(
+                elem, ['ISO 2022 IR 100', 'ISO 2022 IR 146'])
+
+    def test_japanese_multi_byte_personname(self):
+        """Test japanese person name which has multi byte strings are
+        correctly encoded."""
+        file_path = get_charset_files('chrH32.dcm')[0]
+        ds = dcmread(file_path)
+        ds.decode()
+
+        if hasattr(ds.PatientName, 'original_string'):
+            original_string = ds.PatientName.original_string
+            ds.PatientName.original_string = None
+            fp = DicomBytesIO()
+            fp.is_implicit_VR = False
+            fp.is_little_endian = True
+            ds.save_as(fp, write_like_original=False)
+            fp.seek(0)
+            ds_out = dcmread(fp)
+            assert original_string == ds_out.PatientName.original_string
+
+    def test_japanese_multi_byte_encoding(self):
+        """Test japanese multi byte strings are correctly encoded."""
+        encoded = pydicom.charset.encode_string(u'あaｱア',
+                                                ['shift_jis', 'iso2022_jp'])
+        assert b'\x1b$B$"\x1b(Ja\x1b)I\xb1\x1b$B%"\x1b(J' == encoded
+
+    def test_bad_japanese_encoding(self):
+        """Test japanese multi byte strings are not correctly encoded."""
+        with pytest.warns(UserWarning,
+                          match=u"Failed to encode value with encodings"
+                                u": shift_jis - using replacement character"
+                                u"s in encoded string"):
+            encoded = pydicom.charset.encode_string(u'あaｱア', ['shift_jis'])
+            assert b'?a??' == encoded
diff --git a/pydicom/tests/test_dataelem.py b/pydicom/tests/test_dataelem.py
index 4b1df7a49..a5011c867 100644
--- a/pydicom/tests/test_dataelem.py
+++ b/pydicom/tests/test_dataelem.py
@@ -317,6 +317,23 @@ class DataElementTests(unittest.TestCase):
         with pytest.raises(TypeError):
             elem[0]
 
+    def test_repval_large_elem(self):
+        """Test DataElement.repval doesn't return a huge string for a large
+        value"""
+        elem = DataElement(0x00820003, 'UT', 'a'*1000)
+        assert len(elem.repval) < 100
+
+    def test_repval_large_vm(self):
+        """Test DataElement.repval doesn't return a huge string for a large
+        vm"""
+        elem = DataElement(0x00080054, 'AE', 'a\\'*1000+'a')
+        assert len(elem.repval) < 100
+
+    def test_repval_strange_type(self):
+        """Test DataElement.repval doesn't break with bad types"""
+        elem = DataElement(0x00020001, 'OB', 0)
+        assert len(elem.repval) < 100
+
     def test_private_tag_in_repeater_range(self):
         """Test that an unknown private tag (e.g. a tag not in the private
         dictionary) in the repeater range is not handled as a repeater tag
diff --git a/pydicom/tests/test_dataset.py b/pydicom/tests/test_dataset.py
index c5bc064ca..7fd3d09c4 100644
--- a/pydicom/tests/test_dataset.py
+++ b/pydicom/tests/test_dataset.py
@@ -138,30 +138,22 @@ class DatasetTests(unittest.TestCase):
         # messages
         ds = Dataset()
         ds.PatientID = "123456"  # Valid value
-        ds.SmallestImagePixelValue = 0  # Invalid value
+        ds.SmallestImagePixelValue = BadRepr()  # Invalid value
 
-        if compat.in_PyPy:
-            expected_msg = ("With tag (0028, 0106) got exception: "
-                            "'int' has no length")
-        else:
-            expected_msg = ("With tag (0028, 0106) got exception: "
-                            "object of type 'int' has no len()")
+        expected_msg = ("With tag (0028, 0106) got exception: "
+                        "bad repr")
 
-        self.failUnlessExceptionArgs(expected_msg, TypeError, lambda: str(ds))
+        self.failUnlessExceptionArgs(expected_msg, ValueError, lambda: str(ds))
 
     def testTagExceptionWalk(self):
         # When recursing through dataset, a tag number should appear in
         # error messages
         ds = Dataset()
         ds.PatientID = "123456"  # Valid value
-        ds.SmallestImagePixelValue = 0  # Invalid value
+        ds.SmallestImagePixelValue = BadRepr()  # Invalid value
 
-        if compat.in_PyPy:
-            expected_msg = ("With tag (0028, 0106) got exception: "
-                            "'int' has no length")
-        else:
-            expected_msg = ("With tag (0028, 0106) got exception: "
-                            "object of type 'int' has no len()")
+        expected_msg = ("With tag (0028, 0106) got exception: "
+                        "bad repr")
 
         def callback(dataset, data_element):
             return str(data_element)
@@ -169,7 +161,7 @@ class DatasetTests(unittest.TestCase):
         def func(dataset=ds):
             return dataset.walk(callback)
 
-        self.failUnlessExceptionArgs(expected_msg, TypeError, func)
+        self.failUnlessExceptionArgs(expected_msg, ValueError, func)
 
     def dummy_dataset(self):
         # This dataset is used by many of the tests
@@ -235,12 +227,30 @@ class DatasetTests(unittest.TestCase):
 
     def test_pop(self):
         ds = self.dummy_dataset()
+        with pytest.raises(KeyError):
+            ds.pop(0x300a00b244)
         assert 'default' == ds.pop('dummy', 'default')
         elem = ds.pop(0x300a00b2)
         assert 'unit001' == elem.value
         with pytest.raises(KeyError):
             ds.pop(0x300a00b2)
 
+    def test_pop_using_tuple(self):
+        ds = self.dummy_dataset()
+        elem = ds.pop((0x300a, 0x00b2))
+        assert 'unit001' == elem.value
+        with pytest.raises(KeyError):
+            ds.pop((0x300a, 0x00b2))
+
+    def test_pop_using_keyword(self):
+        ds = self.dummy_dataset()
+        with pytest.raises(KeyError):
+            ds.pop('InvalidName')
+        elem = ds.pop('TreatmentMachineName')
+        assert 'unit001' == elem.value
+        with pytest.raises(KeyError):
+            ds.pop('TreatmentMachineName')
+
     def test_popitem(self):
         ds = self.dummy_dataset()
         elem = ds.popitem()
@@ -257,6 +267,33 @@ class DatasetTests(unittest.TestCase):
         assert elem.value == 'Test'
         assert 2 == len(ds)
 
+    def test_setdefault_tuple(self):
+        ds = self.dummy_dataset()
+        elem = ds.setdefault((0x300a, 0x00b2), 'foo')
+        assert 'unit001' == elem.value
+        elem = ds.setdefault((0x0010, 0x0010), DataElement(
+            0x00100010, 'PN', "Test"))
+        assert elem.value == 'Test'
+        assert 2 == len(ds)
+
+    def test_setdefault_use_value(self):
+        ds = self.dummy_dataset()
+        elem = ds.setdefault((0x0010, 0x0010), "Test")
+        assert elem.value == 'Test'
+        assert 2 == len(ds)
+        with pytest.raises(KeyError, match=r'Tag \(0011, 0010\) not found '
+                                           r'in DICOM dictionary'):
+            ds.setdefault((0x0011, 0x0010), "Test")
+
+    def test_setdefault_keyword(self):
+        ds = self.dummy_dataset()
+        elem = ds.setdefault('TreatmentMachineName', 'foo')
+        assert 'unit001' == elem.value
+        elem = ds.setdefault('PatientName',
+                             DataElement(0x00100010, 'PN', "Test"))
+        assert elem.value == 'Test'
+        assert 2 == len(ds)
+
     def testGetExists1(self):
         """Dataset: dataset.get() returns an existing item by name.........."""
         ds = self.dummy_dataset()
@@ -1526,3 +1563,8 @@ class FileDatasetTests(unittest.TestCase):
         di = dict()
         expected_diff = {'__class__', '__doc__', '__hash__'}
         assert expected_diff == set(dir(di)) - set(dir(ds))
+
+
+class BadRepr(object):
+    def __repr__(self):
+        raise ValueError("bad repr")
diff --git a/pydicom/tests/test_numpy_pixel_data.py b/pydicom/tests/test_numpy_pixel_data.py
index dcafe5555..51a13a67f 100644
--- a/pydicom/tests/test_numpy_pixel_data.py
+++ b/pydicom/tests/test_numpy_pixel_data.py
@@ -986,13 +986,23 @@ class TestNumpy_GetPixelData(object):
         # Too short
         ds.PixelData = ds.PixelData[:-1]
         msg = (
-            r"The length of the pixel data in the dataset doesn't match the "
-            r"expected amount \(479999 vs. 480000 bytes\). The dataset may be "
-            r"corrupted or there may be an issue with the pixel data handler."
+            r"The length of the pixel data in the dataset \(479999 bytes\) "
+            r"doesn't match the expected length \(480000 bytes\). "
+            r"The dataset may be corrupted or there may be an issue "
+            r"with the pixel data handler."
         )
         with pytest.raises(ValueError, match=msg):
             get_pixeldata(ds)
 
+    def test_missing_padding_warns(self):
+        """A warning shall be issued if the padding for odd data is missing."""
+        ds = dcmread(EXPL_8_3_1F_ODD)
+        # remove the padding byte
+        ds.PixelData = ds.PixelData[:-1]
+        msg = "The pixel data length is odd and misses a padding byte."
+        with pytest.warns(UserWarning, match=msg):
+            get_pixeldata(ds)
+
     def test_change_photometric_interpretation(self):
         """Test get_pixeldata changes PhotometricInterpretation if required."""
         def to_rgb(ds):