Repository: libsdl-org/SDL (C)

Issue:
[SDL3] surface_testOverflow failing on i386 in older distributions
To reproduce:

* Have chroots/containers for various Debian derivatives (I used various branches of Valve's Steam Runtime on an autobuilder system)
* Install build-dependencies of SDL3: very similar to the packaging from https://tracker.debian.org/pkg/libsdl3, but lightly patched to work with older debhelper versions
* Try to build SDL3, currently a recent but slightly outdated commit 99f6bcf, again with packaging very similar to https://tracker.debian.org/pkg/libsdl3
* During the build, run build-time tests including `testautomation`

Expected result: tests pass

Actual result:

* On Steam Runtime 3 'sniper' (Debian 11 derivative) and newer experimental branches (Debian ≥ 12 derivatives), it works fine
* On Steam Runtime 2 'soldier' (Debian 10 derivative), on x86_64 the test succeeds, but on i386 it fails
* On Steam Runtime 1 'scout' (Ubuntu 12.04 derivative), same as soldier

The end of `surface_testOverflow` looks like this:

```
[  488s] INFO:  01/16/24 15:35:01: Assert '17px * 1 bit per px fits in 3 bytes: (success)': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '17px * 1 bit per px fits in 3 bytes: (success)': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '5px * 8 bits per px fits in 5 bytes: (success)': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '5px * 8 bits per px fits in 5 bytes: (success)': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert 'Should detect pitch < width * bpp': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert 'Expected "Parameter 'pitch' is invalid", got "Parameter 'pitch' is invalid"': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert 'Should detect pitch < width * bpp': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert 'Expected "Parameter 'pitch' is invalid", got "Parameter 'pitch' is invalid"': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '3px * 15 (really 16) bits per px fits in 6 bytes: (success)': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '5px * 15 (really 16) bits per px fits in 6 bytes: (success)': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '4px * 15 (really 16) bits per px doesn't fit in 6 bytes': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert 'Expected "Parameter 'pitch' is invalid", got "Parameter 'pitch' is invalid"': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert '4px * 15 (really 16) bits per px doesn't fit in 6 bytes': [0;32mPassed[0m
[  488s] INFO:  01/16/24 15:35:01: Assert 'Expected "Parameter 'pitch' is invalid", got "Parameter 'pitch' is invalid"': [0;32mPassed[0m
[  488s] ERROR: 01/16/24 15:35:04: Assert 'Should detect overflow in width + alignment': [0;31mFailed[0m
[  488s] ERROR: 01/16/24 15:35:04: Assert 'Expected "Out of memory", got "Parameter 'pitch' is invalid"': [0;31mFailed[0m
[  488s] INFO:  01/16/24 15:35:04: Assert 'Should detect overflow in width * bytes per pixel': [0;32mPassed[0m
[  488s] ERROR: 01/16/24 15:35:04: Assert 'Expected "Out of memory", got "Parameter 'pitch' is invalid"': [0;31mFailed[0m
[  488s] INFO:  01/16/24 15:35:04: Assert 'Should detect overflow in width * height': [0;32mPassed[0m
[  488s] ERROR: 01/16/24 15:35:04: Assert 'Expected "Out of memory", got "Parameter 'pitch' is invalid"': [0;31mFailed[0m
[  488s] INFO:  01/16/24 15:35:04: Assert 'Should detect overflow in width * height * bytes per pixel': [0;32mPassed[0m
[  488s] ERROR: 01/16/24 15:35:04: Assert 'Expected "Out of memory", got "Parameter 'pitch' is invalid"': [0;31mFailed[0m
[  488s] ERROR: 01/16/24 15:35:04: Assert Summary: Total=67 [0;32mPassed=62[0m [0;31mFailed=5[0m
[  488s] INFO:  01/16/24 15:35:04: Total Test runtime: 5.1 sec
[  488s] ERROR: 01/16/24 15:35:04: [0;93m>>> Test 'surface_testOverflow':[0m [0;31mFailed[0m
[  488s] INFO:  01/16/24 15:35:04: Total Suite runtime: 5.1 sec
[  488s] ERROR: 01/16/24 15:35:04: Suite Summary: Total=13 [0;32mPassed=9[0m [0;31mFailed=1[0m [0;94mSkipped=3[0m
```

This is a test-case that I added, so I'll look into what is going on here.


Discussion: An easier reproducer that fails in the same way:

```
$ podman run --rm -it -w $(pwd) -v $(pwd):$(pwd):rw -v $(pwd)/.git:$(pwd)/.git:ro registry.gitlab.steamos.cloud/steamrt/soldier/sdk
# apt update
# apt upgrade
# apt build-dep libsdl2    # close enough!
# /usr/share/cmake/debtoolchainfilegen i386 > i386.txt
# cmake -S . -B _build -DCMAKE_TOOLCHAIN_FILE=$(pwd)/i386.txt -DSDL_TESTS=ON
# make -C _build
# cd _build
# ctest -V    # fails
# export SDL_AUDIO_DRIVER=dummy SDL_VIDEO_DRIVER=dummy
# gdb --args ./test/testautomation --filter surface_testOverflow    # fails
```

CMake in Debian 10 is too old to build SDL3, so this can only matter in environments with a backport of a newer CMake, like the older Steam Runtime branches.
I think this regressed in 447b508a. Allocation functions set the error indicator, but `SDL_CalculateSize()` currently doesn't.
In fact this was mainly a bug in the test-case I added for this overflow checking stuff, in both SDL2 and SDL3.

(Investigating it did expose a minor bug in SDL3's error handling since https://github.com/libsdl-org/SDL/commit/447b508a77b8601d23ca48cf189fe4436ab33572, though.)
I don't really understand why this succeeded in a Debian 11 derivative, but failed in a Debian 10 derivative. Perhaps in newer glibc, a malloc() of more than half of the virtual memory space is artificially prevented from succeeding even if there is in fact enough virtual memory available, to make it harder to construct overflow-based exploits? Or something like that?



Fix (patch):
diff --git a/src/video/SDL_surface.c b/src/video/SDL_surface.c
index 3ca8519c837a5..71f1159a447be 100644
--- a/src/video/SDL_surface.c
+++ b/src/video/SDL_surface.c
@@ -47,28 +47,27 @@ static int SDL_CalculateRGBSize(Uint32 format, size_t width, size_t height, size
 {
     if (SDL_BITSPERPIXEL(format) >= 8) {
         if (SDL_size_mul_overflow(width, SDL_BYTESPERPIXEL(format), pitch)) {
-            return -1;
+            return SDL_SetError("width * bpp would overflow");
         }
     } else {
         if (SDL_size_mul_overflow(width, SDL_BITSPERPIXEL(format), pitch)) {
-            return -1;
+            return SDL_SetError("width * bpp would overflow");
         }
         if (SDL_size_add_overflow(*pitch, 7, pitch)) {
-            return -1;
+            return SDL_SetError("aligning pitch would overflow");
         }
         *pitch /= 8;
     }
     if (!minimal) {
         /* 4-byte aligning for speed */
         if (SDL_size_add_overflow(*pitch, 3, pitch)) {
-            return -1;
+            return SDL_SetError("aligning pitch would overflow");
         }
         *pitch &= ~3;
     }
 
     if (SDL_size_mul_overflow(height, *pitch, size)) {
-        /* Overflow... */
-        return -1;
+        return SDL_SetError("height * pitch would overflow");
     }
 
     return 0;
diff --git a/src/video/SDL_yuv.c b/src/video/SDL_yuv.c
index 63e0236854096..cd9fa8b92cca1 100644
--- a/src/video/SDL_yuv.c
+++ b/src/video/SDL_yuv.c
@@ -72,7 +72,7 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
             /* sz_plane == w * h; */
             size_t s1;
             if (SDL_size_mul_overflow(w, h, &s1) < 0) {
-                return -1;
+                return SDL_SetError("width * height would overflow");
             }
             sz_plane = (int) s1;
         }
@@ -81,15 +81,15 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
             /* sz_plane_chroma == ((w + 1) / 2) * ((h + 1) / 2); */
             size_t s1, s2, s3;
             if (SDL_size_add_overflow(w, 1, &s1) < 0) {
-                return -1;
+                return SDL_SetError("width + 1 would overflow");
             }
             s1 = s1 / 2;
             if (SDL_size_add_overflow(h, 1, &s2) < 0) {
-                return -1;
+                return SDL_SetError("height + 1 would overflow");
             }
             s2 = s2 / 2;
             if (SDL_size_mul_overflow(s1, s2, &s3) < 0) {
-                return -1;
+                return SDL_SetError("width * height would overflow");
             }
             sz_plane_chroma = (int) s3;
         }
@@ -97,11 +97,11 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
         /* sz_plane_packed == ((w + 1) / 2) * h; */
         size_t s1, s2;
         if (SDL_size_add_overflow(w, 1, &s1) < 0) {
-            return -1;
+            return SDL_SetError("width + 1 would overflow");
         }
         s1 = s1 / 2;
         if (SDL_size_mul_overflow(s1, h, &s2) < 0) {
-            return -1;
+            return SDL_SetError("width * height would overflow");
         }
         sz_plane_packed = (int) s2;
     }
@@ -118,10 +118,10 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
             /* dst_size == sz_plane + sz_plane_chroma + sz_plane_chroma; */
             size_t s1, s2;
             if (SDL_size_add_overflow(sz_plane, sz_plane_chroma, &s1) < 0) {
-                return -1;
+                return SDL_SetError("Y + U would overflow");
             }
             if (SDL_size_add_overflow(s1, sz_plane_chroma, &s2) < 0) {
-                return -1;
+                return SDL_SetError("Y + U + V would overflow");
             }
             *size = (int)s2;
         }
@@ -135,11 +135,11 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
             /* pitch == ((w + 1) / 2) * 4; */
            size_t p1, p2;
            if (SDL_size_add_overflow(w, 1, &p1) < 0) {
-               return -1;
+               return SDL_SetError("width + 1 would overflow");
            }
            p1 = p1 / 2;
            if (SDL_size_mul_overflow(p1, 4, &p2) < 0) {
-               return -1;
+               return SDL_SetError("width * 4 would overflow");
            }
            *pitch = p2;
         }
@@ -148,7 +148,7 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
             /* dst_size == 4 * sz_plane_packed; */
             size_t s1;
             if (SDL_size_mul_overflow(sz_plane_packed, 4, &s1) < 0) {
-                return -1;
+                return SDL_SetError("plane * 4 would overflow");
             }
             *size = (int) s1;
         }
@@ -164,22 +164,22 @@ int SDL_CalculateYUVSize(Uint32 format, int w, int h, size_t *size, size_t *pitc
             /* dst_size == sz_plane + sz_plane_chroma + sz_plane_chroma; */
             size_t s1, s2;
             if (SDL_size_add_overflow(sz_plane, sz_plane_chroma, &s1) < 0) {
-                return -1;
+                return SDL_SetError("Y + U would overflow");
             }
             if (SDL_size_add_overflow(s1, sz_plane_chroma, &s2) < 0) {
-                return -1;
+                return SDL_SetError("Y + U + V would overflow");
             }
             *size = (int) s2;
         }
         break;
 
     default:
-        return -1;
+        return SDL_Unsupported();
     }
 
     return 0;
 #else
-    return -1;
+    return SDL_Unsupported();
 #endif
 }