Repository: python-pillow/Pillow

Issue:
Pillow built with libjpeg-turbo 3.0.0 errors with _imaging.so: undefined symbol: jpeg12_write_raw_data 
<!--
Thank you for reporting an issue.

Follow these guidelines to ensure your issue is handled properly.

If you have a ...

1.  General question: consider asking the question on Stack Overflow
    with the python-imaging-library tag:

    * https://stackoverflow.com/questions/tagged/python-imaging-library

    Do not ask a question in both places.

    If you think you have found a bug or have an unexplained exception
    then file a bug report here.

2.  Bug report: include a self-contained, copy-pastable example that
    generates the issue if possible. Be concise with code posted.
    Guidelines on how to provide a good bug report:

    * https://stackoverflow.com/help/mcve

    Bug reports which follow these guidelines are easier to diagnose,
    and are often handled much more quickly.

3.  Feature request: do a quick search of existing issues
    to make sure this has not been asked before.

We know asking good questions takes effort, and we appreciate your time.
Thank you.
-->

### What did you do?

Compile Pillow 10.0.0 with libjpeg-turbo 3.0.0 on Linux

### What did you expect to happen?

compile is fine. But runtime gives the following error

`Error Contents: /usr/lib/python3.11/site-packages/PIL/_imaging.so: undefined symbol: jpeg12_write_raw_data ` 

### What actually happened?

When inspecting the _imaging.so the following symbols jpeg12_ come up as NOTYPE

### What are your OS, Python and Pillow versions?

* OS: LibreELEC 12.0
* Python: 3.11.4
* Pillow: 10.0.0

<!--
Please include **code** that reproduces the issue and whenever possible, an **image** that demonstrates the issue. Please upload images to GitHub, not to third-party file hosting sites. If necessary, add the image to a zip or tar archive.

The best reproductions are self-contained scripts with minimal dependencies. If you are using a framework such as Plone, Django, or Buildout, try to replicate the issue just using Pillow.
-->

```python
code goes here
```



Discussion:
Hmm. I know this isn't happening for every Linux environment with libjpeg-turbo 3.0.0, because our CI is able to compile it without a problem - https://github.com/python-pillow/pillow-wheels/actions/runs/5450902902/jobs/9916621404#step:4:7531

I don't suppose it's possible for you to create a Dockerfile to demonstrate the problem?
Just did a full clean build and here is the output I get:

```
readelf -Ws  build.LibreELEC-Generic.x86_64-12.0-devel/install_pkg/Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imaging.so  | grep jpeg1
    10: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND jpeg12_write_scanlines
    11: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND jpeg12_read_scanlines
    12: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND jpeg12_read_raw_data
    13: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND jpeg12_write_raw_data
   155: 0000000000083e20  4531 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_10x10
   244: 000000000008ec60    55 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_1x1
   305: 000000000008ea80   472 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_2x2
   385: 0000000000082bf0   304 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_3x3
   462: 000000000008e790   751 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_4x4
   508: 000000000007f1b0  6775 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_ifast
   509: 000000000007df50  1455 FUNC    GLOBAL DEFAULT   11 jpeg12_fdct_ifast
   533: 00000000000829e0   515 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_5x5
   625: 0000000000082750   641 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_6x6
   699: 00000000000823b0   925 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_7x7
   745: 000000000007ede0   963 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_float
   852: 0000000000082d20  4338 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_9x9
   884: 0000000000080c30  6016 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_islow
   885: 000000000008c2d0  9405 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_16x16
   886: 000000000007e500  2262 FUNC    GLOBAL DEFAULT   11 jpeg12_fdct_islow
   901: 000000000008a440  7818 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_15x15
   922: 0000000000088920  6931 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_14x14
   944: 0000000000088230  1767 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_13x13
   964: 0000000000086b70  5824 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_12x12
   983: 0000000000084fe0  7050 FUNC    GLOBAL DEFAULT   11 jpeg12_idct_11x11

```
the only references to this function in the build tree (*.h and *.c) are:

```
build/libjpeg-turbo-3.0.0/jpeglib.h:EXTERN(JDIMENSION) jpeg12_write_scanlines(j_compress_ptr cinfo,
build/libjpeg-turbo-3.0.0/jsamplecomp.h:#define _jpeg_write_scanlines  jpeg12_write_scanlines

toolchain/x86_64-libreelec-linux-gnu/sysroot/usr/include/jpeglib.h:EXTERN(JDIMENSION) jpeg12_write_scanlines(j_compress_ptr cinfo,
toolchain/include/jpeglib.h:EXTERN(JDIMENSION) jpeg12_write_scanlines(j_compress_ptr cinfo,

install_pkg/libjpeg-turbo-3.0.0/usr/include/jpeglib.h:EXTERN(JDIMENSION) jpeg12_write_scanlines(j_compress_ptr cinfo,

build/tiff-4.5.1/libtiff/tif_jpeg.c:                    (int)jpeg12_write_scanlines(&sp->cinfo.c, scanlines,

build/libjpeg-turbo-3.0.0/example.c:      /* jpeg12_write_scanlines expects an array of pointers to scanlines.
build/libjpeg-turbo-3.0.0/example.c:      (void)jpeg12_write_scanlines(&cinfo, row_pointer12, 1);
build/libjpeg-turbo-3.0.0/turbojpeg-mp.c:#define _jpeg_write_scanlines  jpeg12_write_scanlines
build/libjpeg-turbo-3.0.0/cjpeg.c:      (void)jpeg12_write_scanlines(&cinfo, src_mgr->buffer12, num_scanlines);

toolchain/x86_64-libreelec-linux-gnu/sysroot/usr/share/doc/libjpeg-turbo/example.c:      /* jpeg12_write_scanlines expects an array of pointers to scanlines.
toolchain/x86_64-libreelec-linux-gnu/sysroot/usr/share/doc/libjpeg-turbo/example.c:      (void)jpeg12_write_scanlines(&cinfo, row_pointer12, 1);

toolchain/share/doc/libjpeg-turbo/example.c:      /* jpeg12_write_scanlines expects an array of pointers to scanlines.
toolchain/share/doc/libjpeg-turbo/example.c:      (void)jpeg12_write_scanlines(&cinfo, row_pointer12, 1);
```
Are these functions coming though because of libtiff?

I can reproduce the error now. It is definitely tiff referencing these functions and not being picked up in Pillow.
Our libjpeg and libtiff builds are static
```
git revert libjpeg_turbo # go back to 2.1.4
s/build libjpeg-turbo
s/build tiff
git reset --hard HEAD^ # back to using 3.0.0
s/build libjpeg-turbo
s/build Pillow
readelf -Ws  build.LibreELEC-Generic.x86_64-12.0-devel/install_pkg/Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imaging.so  | grep jpeg1
# no UND 
```
> Hmm. I know this isn't happening for every Linux environment with libjpeg-turbo 3.0.0, because our CI is able to compile it without a problem - https://github.com/python-pillow/pillow-wheels/actions/runs/5450902902/jobs/9916621404#step:4:7531
> 
> I don't suppose it's possible for you to create a Dockerfile to demonstrate the problem?

The Dockerfile we are using for our builds is the jammy one at https://github.com/LibreELEC/LibreELEC.tv/blob/master/tools/docker/README.md - the master branch has the error since Pillow and libjpeg-turbo were updated earlier this week.
> > Hmm. I know this isn't happening for every Linux environment with libjpeg-turbo 3.0.0, because our CI is able to compile it without a problem - https://github.com/python-pillow/pillow-wheels/actions/runs/5450902902/jobs/9916621404#step:4:7531
> > I don't suppose it's possible for you to create a Dockerfile to demonstrate the problem?
> 
> The Dockerfile we are using for our builds is the jammy one at https://github.com/LibreELEC/LibreELEC.tv/blob/master/tools/docker/README.md - the master branch has the error since Pillow and libjpeg-turbo were updated earlier this week. I’m happy to test on my build server / or dev CI tree.


I've attempted to follow the README, but unsuccessfully - https://github.com/radarhere/Pillow/commit/0f749d95a4fd0129e90a126089dd13d35d1b92f2 gives [`mkdir: cannot create directory`](https://github.com/radarhere/Pillow/actions/runs/5491746529/jobs/10008597978#step:4:1268), and so does https://github.com/radarhere/Pillow/commit/b98cc15fc624994844f93264c639be345b0e8180. Any ideas what I'm doing wrong?
> I've attempted to follow the README, but unsuccessfully - [radarhere@0f749d9](https://github.com/radarhere/Pillow/commit/0f749d95a4fd0129e90a126089dd13d35d1b92f2) gives [`mkdir: cannot create directory`](https://github.com/radarhere/Pillow/actions/runs/5491746529/jobs/10008597978#step:4:1268), and so does [radarhere@b98cc15](https://github.com/radarhere/Pillow/commit/b98cc15fc624994844f93264c639be345b0e8180). Any ideas what I'm doing wrong?

Hi @radarhere - will be the UID of the user in the docker container not matching the UID of the extracted LibreELEC.tv directory
https://github.com/heitbaum/LibreELEC.tv/commit/24ed643200ae7254273b4cd4ae188db3a90d2cfb is what I use where 1001 is your uid. 

For GHA we use the following https://github.com/LibreELEC/actions/blob/54f3eb61e9b12a4440964ede866eac674af95f6c/.github/workflows/make-image.yml#L111 to copy the id from the runner through into the container.

For development in the container I use `docker run --rm --log-driver none -v `pwd`:/build -w /build -it libreelec bash` - then just `make image` or `scripts/build libjpeg` ... etc 
Thanks. That progresses further, but [`make image`](https://github.com/radarhere/Pillow/commit/6524d7b1520d78d1df3d946beff45236fc3a945f) [times out trying to access gmp](https://github.com/radarhere/Pillow/actions/runs/5493239392/jobs/10011284493). [`scripts/build packages/python/graphics/Pillow`](https://github.com/radarhere/Pillow/commit/39e5e482e5e0394ee2e0c8b1f08a86e2df8e52e6) does [the same](https://github.com/radarhere/Pillow/actions/runs/5493264679/jobs/10011325258). I also tried [`scripts/build pillow`](https://github.com/radarhere/Pillow/commit/377f6dbad8e09d4b297517ce21378df469e5f01d), but ["pillow/package.mk does not exist"](https://github.com/radarhere/Pillow/actions/runs/5493245077/jobs/10011293939#step:4:1261).

> the master branch has the error since Pillow and libjpeg-turbo were updated earlier this week.

Is that a CI that is accessible anywhere? https://github.com/LibreELEC/LibreELEC.tv/actions doesn't show anything, and https://github.com/LibreELEC/actions/actions/runs/5482487576 succeeded.
`scripts/build Pillow` is uppercase P. one of the few packages that are uppercase. 

All the CI builds are working fine - just that the resulting _imaging.so has incorrect references. the [gmp](https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz) works from here ... -- guess the Internet.




I've now tried [`scripts/build Pillow`](https://github.com/radarhere/Pillow/commit/090943ab1154f8b7260502f479db9cfc683d0c5b) [but libxslt-v1.1.38.tar.bz2 wasn't found](https://github.com/radarhere/Pillow/actions/runs/5493780619/jobs/10012180877). I think trying to fix LibreELEC's failures to download dependencies is outside of the scope of this issue.

I tried downloading Pillow-10.0.0-cp311-cp311-manylinux_2_28_x86_64.whl from [before](https://github.com/python-pillow/pillow-wheels/actions/runs/5431202691) and [after](https://github.com/python-pillow/pillow-wheels/actions/runs/5450902902) we upgraded to libjpeg-turbo 3.0.0 in our wheels and ran `readelf`, but don't see "write_raw_data" in either.

So if there's not a way for me to reproduce what you're seeing in your project and test changes, and there's no difference before and after on our side, I'm personally out of ways to help here.
Thanks @radarhere - I’ll attempt to understand how _imaging.so is built. (Python not being my strong area) 

I have isolated it down to libtiff.a when built against libjpeg-3.0.0

nm  -g build.LibreELEC-Generic.x86_64-12.0-devel/install_pkg/tiff-4.5.1/usr/lib/libtiff.a

```
…
tif_jpeg_12.c.o:
U fprintf
U getenv
U _GLOBAL_OFFSET_TABLE_
U jpeg12_read_raw_data
U jpeg12_read_scanlines
U jpeg12_write_raw_data
U jpeg12_write_scanlines
…
```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -816,6 +816,15 @@ def build_extensions(self):
 
         libs = self.add_imaging_libs.split()
         defs = []
+        if feature.tiff:
+            libs.append(feature.tiff)
+            defs.append(("HAVE_LIBTIFF", None))
+            if sys.platform == "win32":
+                # This define needs to be defined if-and-only-if it was defined
+                # when compiling LibTIFF. LibTIFF doesn't expose it in `tiffconf.h`,
+                # so we have to guess; by default it is defined in all Windows builds.
+                # See #4237, #5243, #5359 for more information.
+                defs.append(("USE_WIN32_FILEIO", None))
         if feature.jpeg:
             libs.append(feature.jpeg)
             defs.append(("HAVE_LIBJPEG", None))
@@ -830,15 +839,6 @@ def build_extensions(self):
         if feature.imagequant:
             libs.append(feature.imagequant)
             defs.append(("HAVE_LIBIMAGEQUANT", None))
-        if feature.tiff:
-            libs.append(feature.tiff)
-            defs.append(("HAVE_LIBTIFF", None))
-            if sys.platform == "win32":
-                # This define needs to be defined if-and-only-if it was defined
-                # when compiling LibTIFF. LibTIFF doesn't expose it in `tiffconf.h`,
-                # so we have to guess; by default it is defined in all Windows builds.
-                # See #4237, #5243, #5359 for more information.
-                defs.append(("USE_WIN32_FILEIO", None))
         if feature.xcb:
             libs.append(feature.xcb)
             defs.append(("HAVE_XCB", None))


Test patch: