Repository: stb-tester/stb-tester

Issue:
match on single channel images ignores template alpha
In `match` if frame is a single channel array (like from `color_diff`) then alpha in the template is ignored:

https://github.com/stb-tester/stb-tester/blob/71b907aa4a3e5208843cc63dc137d818818ce83c/_stbt/match.py#L354

So this behaves incorrectly:

```py
diff = color_diff(self._frame, "#ffffff")
stbt.match("template-with-alpha.png", frame=diff)
```

It should behave as if `diff` had three channels.

Workaround:

```py
diff = color_diff(self._frame, "#ffffff")
diff = load_image(diff, color_channels=3)
stbt.match("template-with-alpha.png", frame=diff)
```


Discussion:
How to tell if the reference image has an alpha channel?

Option 1: Just load the entire image with `cv2.IMREAD_UNCHANGED` and then convert to grayscale if it doesn't have an alpha channel.

Option 2: Look at the PNG headers.

Then, if the image does have an alpha channel, convert the frame from grayscale to BGR? That would be less optimised, but less risky, than changing `match` to handle reference images with 2 channels (grayscale + alpha).

Option 1 is probably fine (`load_image` is memoized). However out of curiosity I looked at the PNG spec:

## PNG spec

https://www.w3.org/TR/png/#5DataRep:

> ## 5\. Datastream structure
> 
> ### 5.1 Introduction
> 
> The PNG datastream consists of a PNG signature followed by a sequence of chunks.
> 
> ### 5.2 PNG signature
> 
> The first eight bytes of a PNG datastream always contain the following (decimal) values:
> 
>     137 80 78 71 13 10 26 10
> 
> which are (in hexadecimal):
> 
>     89 50 4E 47 0D 0A 1A 0A
> 
> This signature indicates that the remainder of the datastream contains a single PNG image, consisting of a series of chunks beginning with an [IHDR][4] chunk and ending with an [IEND][5] chunk.
> 
> ### 5.3 Chunk layout
> 
> Each chunk consists of three or four fields (see [Figure 10][7]). The meaning of the fields is described in [Table 4][8]. The chunk data field may be empty.
> 
> Figure 10 Chunk parts
> 
> <table id="table51" class="numbered simple"><caption>Table <bdi class="tableno">4</bdi> <span class="table-title">Chunk fields</span></caption><tbody><tr><th>Name</th><th>Description</th></tr><tr><td>Length</td><td>A <a href="https://www.w3.org/TR/png/#dfn-png-four-byte-unsigned-integer" class="internalDFN" data-link-type="dfn" id="ref-for-dfn-png-four-byte-unsigned-integer-1">PNG four-byte unsigned integer</a> giving the number of bytes in the chunk's data field. The length counts <strong>only</strong> the data field, <strong>not</strong> itself, the chunk type, or the <a href="https://www.w3.org/TR/png/#dfn-crc" class="internalDFN" data-link-type="dfn" id="ref-for-dfn-crc-1">CRC</a>. Zero is a valid length. Although encoders and decoders should treat the length as unsigned, its value shall not exceed 2<sup>31</sup>-1 bytes.</td></tr><tr><td>Chunk Type</td><td>A sequence of four bytes defining the chunk type. Each byte of a chunk type is restricted to the decimal values 65 to 90 and 97 to 122. These correspond to the uppercase and lowercase ISO 646 [<cite><a class="bibref" data-link-type="biblio" href="https://www.w3.org/TR/png/#bib-iso646" title="Information technology — ISO 7-bit coded character set for information interchange">ISO646</a></cite>] letters (<code>A</code>-<code>Z</code> and <code>a</code>-<code>z</code>) respectively for convenience in description and examination of PNG datastreams. Encoders and decoders shall treat the chunk types as fixed binary values, not character strings. For example, it would not be correct to represent the chunk type <a class="chunk" href="https://www.w3.org/TR/png/#11IDAT">IDAT</a> by the equivalents of those letters in the UCS 2 character set. Additional naming conventions for chunk types are discussed in <a href="https://www.w3.org/TR/png/#5Chunk-naming-conventions" class="sec-ref"><bdi class="secno">5.4 </bdi>Chunk naming conventions</a>.</td></tr><tr><td>Chunk Data</td><td>The data bytes appropriate to the chunk type, if any. This field can be of zero length.</td></tr><tr><td>CRC</td><td>A four-byte <a href="https://www.w3.org/TR/png/#dfn-crc" class="internalDFN" data-link-type="dfn" id="ref-for-dfn-crc-2">CRC</a> calculated on the preceding bytes in the chunk, including the chunk type field and chunk data fields, but <strong>not</strong> including the length field. The <a href="https://www.w3.org/TR/png/#dfn-crc" class="internalDFN" data-link-type="dfn" id="ref-for-dfn-crc-3">CRC</a> can be used to check for corruption of the data. The <a href="https://www.w3.org/TR/png/#dfn-crc" class="internalDFN" data-link-type="dfn" id="ref-for-dfn-crc-4">CRC</a> is always present, even for chunks containing no data. See <a href="https://www.w3.org/TR/png/#5CRC-algorithm" class="sec-ref"><bdi class="secno">5.5 </bdi>CRC algorithm</a>.</td></tr></tbody></table>
> 
> The chunk data length may be any number of bytes up to the maximum; therefore, implementors cannot assume that chunks are aligned on any boundaries larger than bytes.
> 
> [1]: https://www.w3.org/TR/png/#5DataRep
> [2]: https://www.w3.org/TR/png/#5Introduction
> [3]: https://www.w3.org/TR/png/#5PNG-file-signature
> [4]: https://www.w3.org/TR/png/#11IHDR
> [5]: https://www.w3.org/TR/png/#11IEND
> [6]: https://www.w3.org/TR/png/#5Chunk-layout
> [7]: https://www.w3.org/TR/png/#figure411 "Chunk parts"
> [8]: https://www.w3.org/TR/png/#table51 "Chunk fields"
> [9]: https://www.w3.org/TR/png/#dfn-png-four-byte-unsigned-integer
> [10]: https://www.w3.org/TR/png/#dfn-crc
> [11]: https://www.w3.org/TR/png/#bib-iso646 "Information technology — ISO 7-bit coded character set for information interchange"
> [12]: https://www.w3.org/TR/png/#11IDAT
> [13]: https://www.w3.org/TR/png/#5Chunk-naming-conventions
> [14]: https://www.w3.org/TR/png/#dfn-crc
> [15]: https://www.w3.org/TR/png/#dfn-crc
> [16]: https://www.w3.org/TR/png/#dfn-crc
> [17]: https://www.w3.org/TR/png/#5CRC-algorithm

https://www.w3.org/TR/png/#11IHDR:

> #### 11.2.1 IHDR Image header
> 
> The four-byte chunk type field contains the decimal values
> 
>     73 72 68 82
> 
> The [IHDR][2] chunk shall be the first chunk in the PNG datastream. It contains:
> 
> <table class="numbered simple" id="table-the-ihdr-chunk"><caption>Table <bdi class="tableno">12</bdi> <span class="table-title">The IHDR chunk</span></caption><tbody><tr><th>Name</th><th>Size</th></tr><tr><td>Width</td><td>4 bytes</td></tr><tr><td>Height</td><td>4 bytes</td></tr><tr><td>Bit depth</td><td>1 byte</td></tr><tr><td>Colour type</td><td>1 byte</td></tr><tr><td>Compression method</td><td>1 byte</td></tr><tr><td>Filter method</td><td>1 byte</td></tr><tr><td>Interlace method</td><td>1 byte</td></tr></tbody></table>
> 
> [...]
> 
> [Colour type][6] is a single-byte integer.
> 
> Bit depth restrictions for each [colour type][7] are imposed to simplify implementations and to prohibit combinations that do not compress well. The allowed combinations are defined in [Table 13][8].
> 
> <table id="table111" class="numbered simple"><caption>Table <bdi class="tableno">13</bdi> <span class="table-title">Allowed combinations of <a href="https://www.w3.org/TR/png/#3colourType" class="internalDFN" data-link-type="dfn" id="ref-for-3colourType-10">colour type</a> and bit depth</span></caption><tbody><tr><th>PNG image type</th><th>Colour type</th><th>Allowed bit depths</th><th>Interpretation</th></tr><tr><td>Greyscale</td><td>0</td><td>1, 2, 4, 8, 16</td><td>Each pixel is a greyscale sample</td></tr><tr><td>Truecolour</td><td>2</td><td>8, 16</td><td>Each pixel is an R,G,B triple</td></tr><tr><td>Indexed-colour</td><td>3</td><td>1, 2, 4, 8</td><td>Each pixel is a palette index; a <a class="chunk" href="https://www.w3.org/TR/png/#11PLTE">PLTE</a> chunk shall appear.</td></tr><tr><td>Greyscale with alpha</td><td>4</td><td>8, 16</td><td>Each pixel is a greyscale sample followed by an alpha sample.</td></tr><tr><td>Truecolour with alpha</td><td>6</td><td>8, 16</td><td>Each pixel is an R,G,B triple followed by an alpha sample.</td></tr></tbody></table>
> 
> [1]: https://www.w3.org/TR/png/#11IHDR
> [2]: https://www.w3.org/TR/png/#11IHDR
> [3]: https://www.w3.org/TR/png/#dfn-png-four-byte-unsigned-integer
> [4]: https://www.w3.org/TR/png/#3colourType
> [5]: https://www.w3.org/TR/png/#6Colour-values
> [6]: https://www.w3.org/TR/png/#3colourType
> [7]: https://www.w3.org/TR/png/#3colourType
> [8]: https://www.w3.org/TR/png/#table111 "Allowed combinations of colour type and bit depth"
> [9]: https://www.w3.org/TR/png/#3colourType
> [10]: https://www.w3.org/TR/png/#11PLTE
> [11]: https://www.w3.org/TR/png/#3colourType
> [12]: https://www.w3.org/TR/png/#4Concepts.PNGImage

So colour type should be the 26th byte.

Colour type is a bit-field where "4" means there's an alpha channel.

https://www.w3.org/TR/png/#6Colour-values:

> ### 6.1 Colour types and values
> 
> [][1]
> 
> As explained in [4.5 PNG image][2] there are five types of PNG image. Corresponding to each type is a colour type, which is the sum of the following values: 1 (palette used), 2 ([truecolour][3] used) and 4 (alpha used). Greyscale and [truecolour][4] images may have an explicit alpha channel.
> 
> [1]: https://www.w3.org/TR/png/#6Colour-values
> [2]: https://www.w3.org/TR/png/#4Concepts.PNGImage
> [3]: https://www.w3.org/TR/png/#3truecolour
> [4]: https://www.w3.org/TR/png/#3truecolour

Note that there are other ways of specifying alpha: "Indexed color", where each pixel is an index into a table of colours (which can include alpha); or a separate [tRNS](https://www.w3.org/TR/png/#11tRNS) chunk which can specify that a single RGB value is to be treated as "fully transparent". We can probably ignore both of these.


Resolving patch:
diff --git a/_stbt/match.py b/_stbt/match.py
--- a/_stbt/match.py
+++ b/_stbt/match.py
@@ -20,8 +20,8 @@
 from .config import ConfigurationError, get_config
 from .imgproc_cache import memoize_iterator
 from .imgutils import (
-    crop, FrameT, _frame_repr, ImageT, _image_region, limit_time, load_image,
-    _validate_region)
+    Frame, crop, FrameT, _frame_repr, ImageT, _image_region, limit_time,
+    load_image, _validate_region)
 from .logging import (_Annotation, ddebug, debug, draw_on, get_debug_level,
                       ImageLogger)
 from .types import Position, Region, UITestFailure
@@ -330,6 +330,48 @@ def match_all(
             break
 
 
+def _norm_frame(frame: "FrameT"):
+    """Normalise single channel images to shape (h, w, 3) rather than (h, w) or
+    (h, w, 1).  match has the invariant that it behaves the same as if you'd
+    run `frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2BGR)` first.
+
+    We fiddle with the strides of the numpy array to avoid copying the data.
+    """
+    if not isinstance(frame, numpy.ndarray):
+        raise TypeError(
+            "Expected numpy.ndarray, got %r" % (type(frame).__name__,))
+    if frame.dtype != numpy.uint8:
+        raise TypeError(
+            "Expected numpy.ndarray of uint8, got %r" % (frame.dtype,))
+    frame = frame.view()
+    if frame.ndim == 2:
+        frame.shape = frame.shape + (1,)
+    if frame.ndim != 3:
+        raise ValueError(
+            "Invalid shape for frame: %r. Shape must have 2 or 3 elements" %
+            (frame.shape,))
+    if frame.shape[2] not in (1, 3):
+        raise ValueError(
+            "Invalid shape for frame: %r. Shape must have 1 or 3 channels" %
+            (frame.shape,))
+    if frame.shape[2] == 1:
+        frame = numpy.lib.stride_tricks.as_strided(
+            frame, frame.shape[:2] + (3,), frame.strides[:2] + (0,),
+            subok=True, writeable=False)
+    return frame
+
+
+def test_norm_frame():
+    frame = Frame(
+        numpy.random.randint(256, size=(5, 5, 1), dtype=numpy.uint8), time=56)
+    normed = _norm_frame(frame)
+    assert normed.shape == (5, 5, 3)
+    assert isinstance(normed, Frame)
+    assert normed.time == 56
+    assert numpy.all(normed[:, :, 0] == normed[:, :, 1])
+    assert numpy.all(normed[:, :, 0] == normed[:, :, 2])
+
+
 def _match_all(image, frame, match_parameters, region):
     """
     Generator that yields a sequence of zero or more truthy MatchResults,
@@ -342,16 +384,10 @@ def _match_all(image, frame, match_parameters, region):
         from stbt_core import get_frame
         frame = get_frame()
 
-    # Normalise single channel images to shape (h, w, 1) rather than just (h, w)
-    frame = frame.view()
-    if len(frame.shape) == 2:
-        frame.shape = frame.shape + (1,)
-    if len(frame.shape) != 3:
-        raise ValueError(
-            "Invalid shape for frame: %r. Shape must have 2 or 3 elements" %
-            (frame.shape,))
+    # Normalise single channel images to shape (h, w, 3) rather than just (h, w)
+    frame = _norm_frame(frame)
 
-    t = load_image(image, color_channels={1: 1, 3: (3, 4)}[frame.shape[2]])
+    t = load_image(image, color_channels=(3, 4))
 
     if any(frame.shape[x] < t.shape[x] for x in (0, 1)):
         raise ValueError("Frame %r must be larger than reference image %r"
@@ -359,10 +395,6 @@ def _match_all(image, frame, match_parameters, region):
     if any(t.shape[x] < 1 for x in (0, 1)):
         raise ValueError("Reference image %r must contain some data"
                          % (t.shape,))
-    if (frame.shape[2], t.shape[2]) not in [(1, 1), (3, 3), (3, 4)]:
-        raise ValueError(
-            "Frame %r and reference image %r must have the same number of "
-            "channels" % (frame.shape, t.shape))
 
     if t.shape[2] == 4:
         if cv2_compat.version < [3, 0, 0]:
diff --git a/_stbt/sqdiff.py b/_stbt/sqdiff.py
--- a/_stbt/sqdiff.py
+++ b/_stbt/sqdiff.py
@@ -43,7 +43,7 @@ class _SqdiffResult(ctypes.Structure):
 }
 
 
-def sqdiff(template, frame):
+def sqdiff(template, frame) -> "tuple[int, int]":
     if template.shape[:2] != frame.shape[:2]:
         raise ValueError("Template and frame must be the same size")
     try:


Test patch:
diff --git a/tests/button-grayscale.png b/tests/button-grayscale.png
new file mode 100644
Binary files /dev/null and b/tests/button-grayscale.png differ
diff --git a/tests/test_match.py b/tests/test_match.py
--- a/tests/test_match.py
+++ b/tests/test_match.py
@@ -181,6 +181,16 @@ def test_that_partly_opaque_alpha_is_treated_as_fully_transparent():
     assert numpy.isclose(m1.first_pass_result, m2.first_pass_result)
 
 
+@pytest.mark.parametrize("shape", [
+    (268, 434, 1),
+    (268, 434),
+])
+def test_transparent_reference_image_with_single_channel_frame(shape):
+    f = stbt.load_image("buttons-on-blue-background.png", color_channels=1)
+    f.shape = shape
+    assert stbt.match("button-grayscale.png", frame=f)
+
+
 @pytest.mark.parametrize("frame,image,expected_region", [
     # pylint:disable=bad-whitespace,line-too-long
     ("images/regression/roku-tile-frame.png", "images/regression/roku-tile-selection.png", stbt.Region(x=325, y=145, right=545, bottom=325)),