Repository: pydicom/pydicom

Issue:
Invoking pixel_array truncates valid data if NumberOfFrames is not defined
<!--
If your issue is that the pixel data "looks green/teal", or has strange colors, you probably need to apply a YCbCr to RGB color space conversion using `pydicom.pixel_data_handlers.convert_color_space()`.

See also: https://github.com/pydicom/pydicom/discussions/1577
-->

**Describe the issue**
For my projects, I recently came across a situation in which the file scanner service used to upload documents to our PACS does not break a multipage scan into individual images within a series. Instead, it concatenates the pixel data of each scanned page into a single array in PixelData. Unfortunately, the Rows and Columns tags are set to represent the dimensions of a single scanned page. When calling pixel_array, it warns that it has truncated 66% of the data because of excess padding. As a result, I end up with only the first page of a three page scanned document. I found out about this interaction during testing of a feature I was setting up.

![image](https://github.com/pydicom/pydicom/assets/127312735/b1b6baef-ae5a-42b3-ab29-bc519263d1f6)

I see the following in the numpy data handler
![image](https://github.com/pydicom/pydicom/assets/127312735/246150e7-29eb-45f6-a166-80e0b6d97c99)
get_expected_length() has the following
![image](https://github.com/pydicom/pydicom/assets/127312735/2ad59fec-4862-427f-8d12-42a192c1f4c9)
get_nr_frames() only checks for NumberOfFrames
![image](https://github.com/pydicom/pydicom/assets/127312735/3c1a2c73-50cb-4858-a565-c9a894ffffde)

![image](https://github.com/pydicom/pydicom/assets/127312735/99b41315-18f2-449d-ac8e-730bbf688b0b)

I worked around this issue by computing the frame count as
`page_count = int(len(img_ds.PixelData) / (rows * columns * samples_per_pixel))`
I then adjusted the Rows value to trick pydicom into giving me the full pixel data as a numpy array (convenient for other image manipulations I do).
`img_ds.Rows = rows * page_count`

I know that the scanner is at fault for not recording the NumberOfFrames in the header, but do you think that we could also check if we get an integer ratio between the expected size and the actual size and use that as the result of get_nr_frames() iff NumberOfFrames is not defined?
Maybe we can default `getattr(ds, 'NumberOfFrames', 1)` to `getattr(ds, "NumberOfFrames", computed_integer_frames)`, where `computed_integer_frames = int(len(ds.PixelData) / (rows * columns * samples_per_pixel))`.

If this is a valid idea, I will try to come up with a PR later in the week.
Let me know if this is something I should open a PR for.

**Steps to reproduce**

1. Create a DICOM file.
2. Set Rows and Columns to reflect the length, width of a frame.
3. Add n frames into the PixelData array.
4. Save the DICOM.
5. Load the DICOM and call pixel_array()
6. Observe warning about excess padding.

![image](https://github.com/pydicom/pydicom/assets/127312735/c74a3402-7828-48df-8bac-f632921c78b9)

I do not have an anonymized representative sample at hand right now.


Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/pydicom/pydicom/pull/2065?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pydicom) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 98.24%. Comparing base [(`d707a03`)](https://app.codecov.io/gh/pydicom/pydicom/commit/d707a03abab1e22f98bb46c2d2fb38643da7fb2d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pydicom) to head [(`31fca86`)](https://app.codecov.io/gh/pydicom/pydicom/commit/31fca86d04b3897da6639d02b2f8fcb234426054?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pydicom).


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2065   +/-   ##
=======================================
  Coverage   98.24%   98.24%           
=======================================
  Files          76       76           
  Lines       12422    12434   +12     
=======================================
+ Hits        12204    12216   +12     
  Misses        218      218           
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/pydicom/pydicom/pull/2065?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pydicom).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pydicom).


Patch:
diff --git a/doc/guides/decoding/decoder_options.rst b/doc/guides/decoding/decoder_options.rst
index a757f5f12..a98494536 100644
--- a/doc/guides/decoding/decoder_options.rst
+++ b/doc/guides/decoding/decoder_options.rst
@@ -91,6 +91,15 @@ processing applied after decoding to a NumPy :class:`~numpy.ndarray`:
 Miscellaneous Options
 =====================
 
+The following options may be used with both native (uncompressed) and encapsulated
+(compressed) transfer syntaxes when decoding to a NumPy :class:`~numpy.ndarray` or
+buffer-like object:
+
+* `allow_excess_frames`: :class:`bool` - if ``True`` (default) and `src` contains
+  more frames of data than given by the value of (0028,0008) *Number of Frames* then
+  include those extra frames in the returned data, otherwise return only the number
+  of frames given by *Number of Frames*.
+
 The following options may be used with native (uncompressed) transfer syntaxes
 when decoding to a NumPy :class:`~numpy.ndarray`:
 
diff --git a/doc/release_notes/v3.0.0.rst b/doc/release_notes/v3.0.0.rst
index 3893d247f..df9c3455d 100644
--- a/doc/release_notes/v3.0.0.rst
+++ b/doc/release_notes/v3.0.0.rst
@@ -60,7 +60,7 @@ Changes
   6 or 7 produces incorrect results, so attempting to do so now raises an exception.
   ``pyjpegls`` or ``pylibjpeg`` with ``pylibjpeg-libjpeg`` can be used instead (:issue:`2008`).
 * Using Pillow with JPEG 2000 encoded > 8-bit multi-sample data (such as RGB) now raises an
-  exception as Pillow cannot decode such data correctly (:issue:`2006`, :issue:`2059`)
+  exception as Pillow cannot decode such data correctly (:issue:`2006`)
 
 
 Removals
@@ -176,6 +176,8 @@ Enhancements
     and many others).
   * Returning excess frames for JPEG when there is no Basic or Extended Offset
     Table and the *Number of Frames* is incorrect (:issue:`1666`).
+  * Returning excess frames for native encoding when the *Number of Frames* is
+    incorrect (:issue:`2035`)
   * Returning the decoded pixel data as either a NumPy :class:`~numpy.ndarray` or
     `buffer-like object <https://docs.python.org/3/c-api/buffer.html#bufferobjects>`_.
   * Iterating through either all or specific frames.
diff --git a/src/pydicom/pixels/decoders/base.py b/src/pydicom/pixels/decoders/base.py
index f633ae90d..75490c9a9 100644
--- a/src/pydicom/pixels/decoders/base.py
+++ b/src/pydicom/pixels/decoders/base.py
@@ -67,6 +67,8 @@ class DecodeOptions(RunnerOptions, total=False):
 
     # The VR used for the pixel data - may be used with Explicit VR Big Endian
     pixel_vr: str
+    # Allow returning/yielding excess frames when found
+    allow_excess_frames: bool
 
     ## Native transfer syntax decoding options
     # Return/yield a view of the original buffer where possible
@@ -202,6 +204,7 @@ class DecodeRunner(RunnerBase):
         self._opts: DecodeOptions = {
             "transfer_syntax_uid": tsyntax,
             "as_rgb": True,
+            "allow_excess_frames": True,
         }
         self._undeletable = ("transfer_syntax_uid", "pixel_keyword")
         self._decoders: dict[str, DecodeFunction] = {}
@@ -689,7 +692,8 @@ class DecodeRunner(RunnerBase):
     def _validate_buffer(self) -> None:
         """Validate the supplied buffer data."""
         # Check that the actual length of the pixel data is as expected
-        expected = self.frame_length(unit="bytes") * self.number_of_frames
+        frame_length = self.frame_length(unit="bytes")
+        expected = frame_length * self.number_of_frames
         actual = len(cast(Buffer, self._src))
 
         if self.transfer_syntax.is_encapsulated:
@@ -725,6 +729,22 @@ class DecodeRunner(RunnerBase):
                         "changed to either 'RGB' or 'YBR_FULL'"
                     )
 
+            # Determine if there's sufficient padding to contain extra frames
+            elif self.get_option("allow_excess_frames", False):
+                whole_frames = actual // frame_length
+                if whole_frames > self.number_of_frames:
+                    warn_and_log(
+                        "The number of bytes of pixel data is sufficient to contain "
+                        f"{whole_frames} frames which is larger than the given "
+                        f"(0028,0008) 'Number of Frames' value of {self.number_of_frames}. "
+                        "The returned data will include these extra frames and if it's "
+                        "correct then you should update 'Number of Frames' accordingly, "
+                        "otherwise pass 'allow_excess_frames=False' to return only "
+                        f"the first {self.number_of_frames} frames."
+                    )
+                    self.set_option("number_of_frames", whole_frames)
+                    return
+
             # PS 3.5, Section 8.1.1
             warn_and_log(
                 f"The pixel data is {actual} bytes long, which indicates it "
@@ -980,18 +1000,25 @@ class Decoder(CoderBase):
 
         # Check to see if we have any more frames available
         #   Should only apply to JPEG transfer syntaxes
-        excess = []
-        for frame in frame_generator:
-            if len(frame) == runner.frame_length(unit="bytes"):
-                excess.append(np.frombuffer(frame, runner.pixel_dtype))
-                runner.set_option("number_of_frames", runner.number_of_frames + 1)
-
-        if excess:
-            warn_and_log(
-                "More frames have been found in the encapsulated pixel data "
-                "than expected from the supplied number of frames"
-            )
-            arr = np.concatenate([arr, *excess])
+        if runner.get_option("allow_excess_frames", False):
+            excess = []
+            original_nr_frames = runner.number_of_frames
+            for frame in frame_generator:
+                if len(frame) == runner.frame_length(unit="bytes"):
+                    excess.append(np.frombuffer(frame, runner.pixel_dtype))
+                    runner.set_option("number_of_frames", runner.number_of_frames + 1)
+
+            if excess:
+                warn_and_log(
+                    f"{len(excess) + original_nr_frames} frames have been found in "
+                    "the encapsulated pixel data, which is larger than the given "
+                    f"(0028,0008) 'Number of Frames' value of {original_nr_frames}. "
+                    "The returned data will include these extra frames and if it's "
+                    "correct then you should update 'Number of Frames' accordingly, "
+                    "otherwise pass 'allow_excess_frames=False' to return only the "
+                    f"first {original_nr_frames} frames."
+                )
+                arr = np.concatenate([arr, *excess])
 
         runner.set_option("bits_allocated", original_bits_allocated)
 
@@ -1280,19 +1307,26 @@ class Decoder(CoderBase):
 
         # Check to see if we have any more frames available
         #   Should only apply to JPEG transfer syntaxes
-        excess = []
-        for frame in frame_generator:
-            if len(frame) == runner.frame_length(unit="bytes"):
-                excess.append(frame)
-                runner.set_option("number_of_frames", runner.number_of_frames + 1)
-                bits_allocated.append(runner.bits_allocated)
-
-        if excess:
-            warn_and_log(
-                "More frames have been found in the encapsulated pixel data "
-                "than expected from the supplied number of frames"
-            )
-            frames.extend(excess)
+        if runner.get_option("allow_excess_frames", False):
+            excess = []
+            original_nr_frames = runner.number_of_frames
+            for frame in frame_generator:
+                if len(frame) == runner.frame_length(unit="bytes"):
+                    excess.append(frame)
+                    runner.set_option("number_of_frames", runner.number_of_frames + 1)
+                    bits_allocated.append(runner.bits_allocated)
+
+            if excess:
+                warn_and_log(
+                    f"{len(excess) + len(frames)} frames have been found in the "
+                    "encapsulated pixel data, which is larger than the given "
+                    f"(0028,0008) 'Number of Frames' value of {original_nr_frames}. "
+                    "The returned data will include these extra frames and if it's "
+                    "correct then you should update 'Number of Frames' accordingly, "
+                    "otherwise pass 'allow_excess_frames=False' to return only the "
+                    f"first {original_nr_frames} frames."
+                )
+                frames.extend(excess)
 
         # Each frame may have been encoded using a different precision. On
         #   decoding this may result in different container sizes per frame


Test patch:
diff --git a/tests/pixels/test_decoder_base.py b/tests/pixels/test_decoder_base.py
index c999ea820..c9919dbfd 100644
--- a/tests/pixels/test_decoder_base.py
+++ b/tests/pixels/test_decoder_base.py
@@ -7,7 +7,7 @@ from sys import byteorder
 
 import pytest
 
-from pydicom import config
+from pydicom import config, dcmread
 from pydicom.dataset import Dataset
 from pydicom.encaps import get_frame, generate_frames, encapsulate
 from pydicom.pixels import get_decoder
@@ -166,6 +166,7 @@ class TestDecodeRunner:
             "Options\n"
             "  transfer_syntax_uid: 1.2.840.10008.1.2.5\n"
             "  as_rgb: True\n"
+            "  allow_excess_frames: True\n"
             "  pixel_keyword: PixelData\n"
             "Decoders\n"
             "  foo"
@@ -1096,6 +1097,33 @@ class TestDecoder_Array:
         assert arr.dtype == reference.dtype
         assert not arr.flags.writeable  # read-only
 
+    def test_native_excess_frames(self):
+        """Test returning excess frame data"""
+        decoder = get_decoder(ExplicitVRLittleEndian)
+        ds = dcmread(EXPL_16_1_10F.path)
+        ds.NumberOfFrames = 9
+
+        msg = (
+            "The number of bytes of pixel data is sufficient to contain 10 frames "
+            r"which is larger than the given \(0028,0008\) 'Number of Frames' "
+            "value of 9. The returned data will include these extra frames and if "
+            "it's correct then you should update 'Number of Frames' accordingly, "
+            "otherwise pass 'allow_excess_frames=False' to return only the first "
+            "9 frames."
+        )
+        with pytest.warns(UserWarning, match=msg):
+            arr, meta = decoder.as_array(ds)
+
+        assert arr.shape == (10, 64, 64)
+        assert meta["number_of_frames"] == 10
+
+        msg = "contains 8192 bytes of excess padding"
+        with pytest.warns(UserWarning, match=msg):
+            arr, meta = decoder.as_array(ds, allow_excess_frames=False)
+
+        assert arr.shape == (9, 64, 64)
+        assert meta["number_of_frames"] == 9
+
     def test_encapsulated_index(self):
         """Test `index` with an encapsulated pixel data."""
         decoder = get_decoder(RLELossless)
@@ -1135,8 +1163,11 @@ class TestDecoder_Array:
         runner.set_source(reference.ds)
 
         msg = (
-            "More frames have been found in the encapsulated pixel data than "
-            "expected from the supplied number of frames"
+            "11 frames have been found in the encapsulated pixel data, which is "
+            r"larger than the given \(0028,0008\) 'Number of Frames' value of 10. "
+            "The returned data will include these extra frames and if it's correct "
+            "then you should update 'Number of Frames' accordingly, otherwise pass "
+            "'allow_excess_frames=False' to return only the first 10 frames."
         )
         with pytest.warns(UserWarning, match=msg):
             arr, meta = decoder.as_array(src, **runner.options)
@@ -1144,6 +1175,11 @@ class TestDecoder_Array:
         assert arr.shape == (11, 64, 64)
         assert meta["number_of_frames"] == 11
 
+        runner.set_option("allow_excess_frames", False)
+        arr, meta = decoder.as_array(src, **runner.options)
+        assert arr.shape == (10, 64, 64)
+        assert meta["number_of_frames"] == 10
+
     def test_processing_colorspace(self):
         """Test the processing colorspace options."""
         decoder = get_decoder(ExplicitVRLittleEndian)
@@ -1562,8 +1598,11 @@ class TestDecoder_Buffer:
         runner.set_source(reference.ds)
 
         msg = (
-            "More frames have been found in the encapsulated pixel data than "
-            "expected from the supplied number of frames"
+            "11 frames have been found in the encapsulated pixel data, which is "
+            r"larger than the given \(0028,0008\) 'Number of Frames' value of 10. "
+            "The returned data will include these extra frames and if it's correct "
+            "then you should update 'Number of Frames' accordingly, otherwise pass "
+            "'allow_excess_frames=False' to return only the first 10 frames."
         )
         with pytest.warns(UserWarning, match=msg):
             buffer, meta = decoder.as_buffer(src, **runner.options)
@@ -1571,6 +1610,11 @@ class TestDecoder_Buffer:
         assert len(buffer) == 11 * 64 * 64 * 2
         assert meta["number_of_frames"] == 11
 
+        runner.set_option("allow_excess_frames", False)
+        buffer, meta = decoder.as_buffer(src, **runner.options)
+        assert len(buffer) == 10 * 64 * 64 * 2
+        assert meta["number_of_frames"] == 10
+
     def test_expb_ow_index_invalid_raises(self):
         """Test invalid index with BE swapped OW raises"""
         decoder = get_decoder(ExplicitVRBigEndian)