Repository: spack/spack

Issue:
mesa18@18.3.6 build fails: AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
### Steps to reproduce the issue

`mesa18@18.3.6` build fails using:
* `spack@develop` (997dba8ffdc01d50dff3ebae791aff3d0145c818 from `Mon Jan 3 18:22:48 2022 +0100`)
* Ubuntu 20.04, x86_64
* GCC 9.3.0

Concrete spec: [mesa18-d7eiib.spec.yaml.txt](https://github.com/spack/spack/files/7804161/mesa18-d7eiib.spec.yaml.txt)

Concretization:
```
 -   mesa18@18.3.6%gcc@9.3.0+glx~llvm~opengles+osmesa patches=55a5611ca9fcbe8324c4d68a07b338134954ff12c5b122dc78ff376f012a1414 swr=none arch=linux-ubuntu20.04-x86_64
[+]      ^autoconf@2.69%gcc@9.3.0 patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee arch=linux-ubuntu20.04-x86_64
[+]          ^m4@1.4.19%gcc@9.3.0+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-ubuntu20.04-x86_64
[+]              ^libsigsegv@2.13%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^perl@5.34.0%gcc@9.3.0+cpanm+shared+threads arch=linux-ubuntu20.04-x86_64
[+]              ^berkeley-db@18.1.40%gcc@9.3.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu20.04-x86_64
[+]              ^bzip2@1.0.8%gcc@9.3.0~debug~pic+shared arch=linux-ubuntu20.04-x86_64
[+]                  ^diffutils@3.8%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]                      ^libiconv@1.16%gcc@9.3.0 libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]              ^gdbm@1.19%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]                  ^readline@8.1%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]                      ^ncurses@6.2%gcc@9.3.0~symlinks+termlib abi=none arch=linux-ubuntu20.04-x86_64
[+]                          ^pkgconf@1.8.0%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-x86_64
[+]      ^automake@1.16.3%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^binutils@2.36.1%gcc@9.3.0~gas+gold+headers~interwork+ld+libiberty~lto~nls+plugins libs=shared,static patches=a51b7bf4607645235be01683a3e2deb3a87d767cc124ea4ef6febf9f32abf4a7 arch=linux-ubuntu20.04-x86_64
[+]      ^bison@3.8.2%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^expat@2.4.1%gcc@9.3.0+libbsd arch=linux-ubuntu20.04-x86_64
[+]          ^libbsd@0.11.3%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^libmd@1.0.3%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^flex@2.6.3%gcc@9.3.0+lex~nls arch=linux-ubuntu20.04-x86_64
[+]          ^findutils@4.8.0%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^gettext@0.21%gcc@9.3.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-ubuntu20.04-x86_64
[+]          ^libxml2@2.9.12%gcc@9.3.0~python arch=linux-ubuntu20.04-x86_64
[+]              ^xz@5.2.5%gcc@9.3.0+pic libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]          ^tar@1.34%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^glproto@1.4.17%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^util-macros@1.19.3%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^libtool@2.4.6%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^libx11@1.7.0%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^inputproto@2.3.2%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^kbproto@1.0.7%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^libxcb@1.14%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^libpthread-stubs@0.4%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^libxau@1.0.8%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]                  ^xproto@7.0.31%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^libxdmcp@1.1.2%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^xcb-proto@1.14.1%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^xextproto@7.3.0%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^xtrans@1.3.5%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^libxext@1.3.3%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]      ^py-mako@1.1.5%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]          ^py-markupsafe@2.0.1%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]              ^py-setuptools@59.4.0%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
[+]                  ^python@3.9.9%gcc@9.3.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4 arch=linux-ubuntu20.04-x86_64
[+]                      ^libffi@3.3%gcc@9.3.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=linux-ubuntu20.04-x86_64
[+]                      ^openssl@1.1.1l%gcc@9.3.0~docs certs=system arch=linux-ubuntu20.04-x86_64
[+]                      ^sqlite@3.36.0%gcc@9.3.0+column_metadata+fts~functions~rtree arch=linux-ubuntu20.04-x86_64
[+]                      ^util-linux-uuid@2.36.2%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
```

Error reproduced with entirely cached dependencies using:
* Docker container image = `ecpe4s/ubuntu20.04-runner-x86_64:2021-11-01`

```
$> spack mirror add E4S https://cache.e4s.io
$> spack buildcache keys -it

$> spack install --cache-only --include-build-deps --only dependencies -f ./mesa18-d7eiib.spec.yaml
OK

$> spack install --no-cache -f ./mesa18-d7eiib.spec.yaml
...
==> Installing mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q
==> Applied patch /spack/var/spack/repos/builtin/packages/mesa18/autotools-x11-nodri.patch
==> mesa18: Executing phase: 'autoreconf'
==> mesa18: Executing phase: 'configure'
==> mesa18: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16' 'V=1'

21 errors found in build log:
     13     libtoolize: copying file 'm4/ltsugar.m4'
     14     libtoolize: copying file 'm4/ltversion.m4'
     15     libtoolize: copying file 'm4/lt~obsolete.m4'
     16     autoreconf: running: /spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_
            path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholde/linux-ubuntu20.04-x86_64/gcc-9.3.0/autoconf-2.69-4wefch2dqfia5xr4oxseyr73xo7p7dav/bi
            n/autoconf --force
     17     autoreconf: configure.ac: not using Autoheader
     18     autoreconf: running: automake --add-missing --copy --force-missing
  >> 19     configure.ac:61: installing 'bin/ar-lib'
  >> 20     configure.ac:61: installing 'bin/compile'
  >> 21     configure.ac:45: installing 'bin/config.guess'
  >> 22     configure.ac:45: installing 'bin/config.sub'
  >> 23     configure.ac:46: installing 'bin/install-sh'
  >> 24     configure.ac:46: installing 'bin/missing'
     25     src/Makefile.am: installing 'bin/depcomp'
     26     parallel-tests: installing 'bin/test-driver'
     27     autoreconf: Leaving directory `.'
     28     ==> mesa18: Executing phase: 'configure'
     29     ==> [2022-01-03-13:40:09.388008] '/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/configure' '--prefix=/spack/opt/spack/[padded-to-256-cha
            rs]/linux-ubuntu20.04-x86_64/gcc-9.3.0/mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q' 'LDFLAGS=-L/spack/opt/spack/[padded-to-256-chars]/linux-ubuntu20.04-x86_64/gcc-9.3.0/ncurses-6
            .2-nxjjphalwbf2ywlxpn74nsfcsrq2bbtd/lib' '--enable-shared' '--disable-static' '--disable-libglvnd' '--disable-nine' '--disable-omx-bellagio' '--disable-omx-tizonia' '--disable-openc
            l' '--disable-opencl-icd' '--disable-va' '--disable-vdpau' '--disable-xa' '--disable-xvmc' '--disable-osmesa' '--with-vulkan-drivers=' '--disable-egl' '--disable-gbm' '--disable-dri
            ' '--enable-opengl' '--enable-gallium-osmesa' '--enable-glx=gallium-xlib' '--disable-gles1' '--disable-gles2' '--enable-shared-glapi' '--disable-llvm' '--with-platforms=x11' '--with
            -gallium-drivers=swrast' '--with-dri-drivers='
     30     checking build system type... x86_64-pc-linux-gnu

     ...

     670        self.process_element(file_name, doc)
     671      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./../../../../src/mapi/glapi/gen/gl_XML.py", line 914, in proce
            ss_element
     672        self.process_OpenGLAPI(file_name, element)
     673      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./../../../../src/mapi/glapi/gen/gl_XML.py", line 919, in proce
            ss_OpenGLAPI
     674        for child in element.getchildren():
     675    AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
  >> 676    make[3]: *** [Makefile:939: ../../../../src/mesa/main/marshal_generated.c] Error 1
     677    make[3]: *** Waiting for unfinished jobs....
     678    Traceback (most recent call last):
     679      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./gl_table.py", line 207, in <module>
     680        main()
     681      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./gl_table.py", line 196, in main
     682        api = gl_XML.parse_GL_API(args.file_name)

     ...

     686        self.process_element(file_name, doc)
     687      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/gl_XML.py", line 914, in process_element
     688        self.process_OpenGLAPI(file_name, element)
     689      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/gl_XML.py", line 919, in process_OpenGLAPI
     690        for child in element.getchildren():
     691    AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
  >> 692    make[3]: *** [Makefile:942: ../../../../src/mesa/main/marshal_generated.h] Error 1
     693    Traceback (most recent call last):
     694      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./gl_table.py", line 207, in <module>
     695    Traceback (most recent call last):
     696      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./gl_procs.py", line 190, in <module>
     697        main()
     698      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./gl_table.py", line 196, in main

     ...

     771        for child in element.getchildren():
     772    AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
     773        self.process_OpenGLAPI(file_name, element)
     774      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/gl_XML.py", line 919, in process_OpenGLAPI
     775        for child in element.getchildren():
     776    AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
  >> 777    make[3]: *** [Makefile:904: ../../../../src/mapi/glapi/glapi_mapi_tmp.h] Error 1
  >> 778    make[3]: *** [Makefile:945: ../../../../src/mesa/main/dispatch.h] Error 1
  >> 779    make[3]: *** [Makefile:914: ../../../../src/mapi/glapi/glapitable.h] Error 1
  >> 780    make[3]: *** [Makefile:948: ../../../../src/mesa/main/remap_helper.h] Error 1
  >> 781    make[3]: *** [Makefile:908: ../../../../src/mapi/glapi/glprocs.h] Error 1
  >> 782    make[3]: *** [Makefile:925: ../../../../src/mapi/glapi/glapi_x86-64.S] Error 1
  >> 783    make[3]: *** [Makefile:936: ../../../../src/mesa/main/api_exec.c] Error 1
  >> 784    make[3]: *** [Makefile:911: ../../../../src/mapi/glapi/glapitemp.h] Error 1
     785    Traceback (most recent call last):
     786      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./glX_proto_size.py", line 699, in <module>
     787        main()
     788      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/./glX_proto_size.py", line 693, in main
     789        api = gl_XML.parse_GL_API(args.filename, glX_XML.glx_item_factory())
     790      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/gl_XML.py", line 44, in parse_GL_API

     ...

     849        self.process_OpenGLAPI(file_name, element)
     850      File "/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen/gl_XML.py", line 919, in process_OpenGLAPI
     851        for child in element.getchildren():
     852    AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
     853        for child in element.getchildren():
     854    AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
  >> 855    make[3]: *** [Makefile:960: ../../../../src/glx/indirect_init.c] Error 1
  >> 856    make[3]: *** [Makefile:957: ../../../../src/glx/indirect.h] Error 1
     857    make[3]: Leaving directory '/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src/mapi/glapi/gen'
  >> 858    make[2]: *** [Makefile:895: all-recursive] Error 1
     859    make[2]: Leaving directory '/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src'
  >> 860    make[1]: *** [Makefile:680: all] Error 2
     861    make[1]: Leaving directory '/tmp/root/spack-stage/spack-stage-mesa18-18.3.6-3kdtnixd32klhqamkwvzaveg5pj6ue5q/spack-src/src'
  >> 862    make: *** [Makefile:694: all-recursive] Error 1
```

### Information on your system

* **Spack:** 0.17.1-697-997dba8ffd
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-cascadelake
* **Concretizer:** clingo

### Additional information

[spack-configure-args.txt](https://github.com/spack/spack/files/7804175/spack-configure-args.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/7804176/spack-build-env.txt)
[spack-build-02-configure-out.txt](https://github.com/spack/spack/files/7804177/spack-build-02-configure-out.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/7804171/spack-build-out.txt)

@ChristianTackeGSI 
@chuckatkins
@v-dobrev

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
We are facing the same build failure while working on the Spack recipe update for ROCm 4.5.2.
Is there any workaround available to complete the build temporarily in order to move on the other the activities on the upstream code while debugging this issue?
> We are facing the same build failure while working on the Spack recipe update for ROCm 4.5.2. Is there any workaround available to complete the build temporarily in order to move on the other the activities on the upstream code while debugging this issue?

What version of `python` are you using? I ran into this myself and while hunting around for a cause I found [references](https://github.com/ocrmypdf/OCRmyPDF/issues/583) to other packages running into this same error. In their analysis, the trouble was that the "getchildren" attribute was deprecated sometime in `python 3.9`. I ended up rolling back to `python 3.8.12` and had no further trouble.
> I ended up rolling back to `python 3.8.12` and had no further trouble.

This worked for me.

I will create a PR adding this requirement to `mesa18` package.py, unless anyone has a better way of handling this?

https://github.com/spack/spack/pull/28345
> > I ended up rolling back to `python 3.8.12` and had no further trouble.
> 
> This worked for me.
> 
> I will create a PR adding this requirement to `mesa18` package.py, unless anyone has a better way of handling this?
> 
> #28345

I think that's the best solution. Thanks!
Thank you @eugeneswalker. Build completed with python@3.8.12
> For your case, you said you see the same problem with /usr/bin/python3
> version 3.6.8 on RH 7.x, are you sure that is the python version being
> used to build Mesa18? Or, is it just the python version driving Spack?

@eugeneswalker I see your point, but no, that's not exactly the problem.
Actually, it looks like it's finding an old python 2.7.

This is on RH 7.9.  I ran `spack external find python` which produced:

```
packages:
  python:
    externals:
    - spec: python@2.7.5+bz2+ctypes+dbm+lzma+nis+pyexpat+readline+sqlite3+ssl~tix~tkinter+uuid+zlib
      prefix: /usr
    - spec: python@3.6.8+bz2+ctypes+dbm+lzma+nis+pyexpat+readline+sqlite3+ssl~tix~tkinter+uuid+zlib
      prefix: /usr
```

So, `/usr/bin/python` is 2.7.5 and `python3` is 3.6.8.

I erased all the installed packages and started from scratch.
`spack spec` says it's using 3.6.8.

```
$ spack spec mesa18~llvm
mesa18@18.3.6%gcc@8.5.0+glx~llvm~opengles+osmesa patches=55a5611ca9fcbe8324c4d68a07b338134954ff12c5b122dc78ff376f012a1414 swr=none arch=linux-rhel7-broadwell
...
    ^py-mako@1.1.5%gcc@8.5.0 arch=linux-rhel7-broadwell
        ^py-markupsafe@2.0.1%gcc@8.5.0 arch=linux-rhel7-broadwell
            ^py-setuptools@59.4.0%gcc@8.5.0 arch=linux-rhel7-broadwell
                ^python@3.6.8%gcc@8.5.0+bz2+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=c129e34472ee39b1083c38a49ad06a8573d8b3b208743a120ca4c0818cdf1801 arch=linux-rhel7-broadwell
```

And indeed, before it crashed, it did not build python 3.9.

But `spack install -v` reveals that mesa was trying to use python 2.7,
and apparently 2.7 doesn't have py-mako.

This is from configure:

```
==> mesa18: Executing phase: 'configure'
==> [2022-01-11-15:07:09.756286] '/home/krentel/elf/spack-repo/build-stage/spack-stage-mesa18-18.3.6-zfbm7kypbulmftdvertfu76ggfwwbyur/spack-src/configure' '--prefix=/home/krentel/elf/install/linux-rhel7-broadwell/gcc-8.5.0/mesa18-18.3.6-zfbm7kypbulmftdvertfu76ggfwwbyur' 'LDFLAGS=-L/home/krentel/elf/install/linux-rhel7-broadwell/gcc-8.5.0/ncurses-6.2-7mumcnj6isssyj4ytpgsky4xv6jwncpq/lib' '--enable-shared' '--disable-static' '--disable-libglvnd' '--disable-nine' '--disable-omx-bellagio' '--disable-omx-tizonia' '--disable-opencl' '--disable-opencl-icd' '--disable-va' '--disable-vdpau' '--disable-xa' '--disable-xvmc' '--disable-osmesa' '--with-vulkan-drivers=' '--disable-egl' '--disable-gbm' '--disable-dri' '--enable-opengl' '--enable-gallium-osmesa' '--enable-glx=gallium-xlib' '--disable-gles1' '--disable-gles2' '--enable-shared-glapi' '--disable-llvm' '--with-platforms=x11' '--with-gallium-drivers=swrast' '--with-dri-drivers='
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
...
checking for GNU make... make
checking for a Python interpreter with version >= 2.7... python
checking for python... /usr/bin/python
checking for python version... 2.7
checking for python platform... linux2
checking for python script directory... ${prefix}/lib/python2.7/site-packages
checking for python extension module directory... ${exec_prefix}/lib64/python2.7/site-packages
checking for a sed that does not truncate output... /usr/bin/sed
...
checking if flex is the lexer generator... yes
checking for indent... indent
checking if module mako in python is installed... No module named mako
no
configure: error: Python mako module v0.8.0 or higher not found
```

So, why is mesa picking up python 2.7 instead of 3.6 ?
Does spack need to be more explicit in telling mesa what python to use ?
I tried another experiment.  I removed python 2.7 from `packages.yaml`
and changed the python dependency to:

```
depends_on('python@3.6.0:3.8.12', type='build')
```

and I still get the same failure with mesa trying to use 2.7.

I've never looked at building mesa18 directly.  But I think this
identifies the solution: spack needs to explicitly tell mesa18 which
python to use and not let mesa search for itself.
Yep, separate from the python <= 3.8.12 issue, there is another
problem of passing the right python + mako to mesa.

In my case, I was using an "external" python3 from /usr/bin.  But
since it's /usr/bin, spack can't put that directory "ahead" of
/usr/bin/python 2.7 on my path.

Mesa configure has a configure variable PYTHON to set the path to the
desired python.  But I haven't yet found a satisfactory way of setting
the path to mako.

@adamjstewart You seem to make a lot of python commits.
Would you like to help us out here?
The problem here is that spack is not passing to mesa18 the locations
of python and mako.
I wonder if any of the `mesa18` maintainers could help us with a config option for specifying the precise python to be used for the build. 
Does mesa have a CMake option like `PYTHON_EXECUTABLE`? If so, the solution is to simply add:
```python
args.append(self.define('PYTHON_EXECUTABLE', self.spec['python'].command.path))
```
The CMake will be forced to use `/usr/bin/python3` instead of `/usr/bin/python2`.
@adamjstewart mesa18 uses autotools, we think that's the reason it's a separate
package.

You can specify PYTHON in mesa18, which works to identify the python
path, but for me, the detection of py-mako still failed, and I don't
see a knob for py-make.

There are a lot of moving parts and I need to walk through the steps
again carefully to identify exactly the bounds of what still fails.
In that case we should definitely set `PYTHON`. For `py-mako`/`py-make`, I wonder if that's fixed by #28346. If not, seeing the `spack-build-env.txt` would be helpful. I want to see what is getting added to `PYTHONPATH` and where the Python libraries actually live.
@adamjstewart  @eugeneswalker 
I ran some more tests, with and without Adam's patch from #28346.
The short answer is that I think #28346 contains the solution to my
problem, and I finally understand why.

This is on RH 8.x.  There is python2 and python3, but no python
symlink, at least not in /usr/bin.  My packages.yaml has python3 from
/usr/bin which is what spack spec and install use.

```
packages:
  python:
    externals:
    - spec: python@3.6.8+bz2+ctypes+dbm+lzma+nis+pyexpat+readline+sqlite3+ssl~tix~tkinter+uuid+zlib
      prefix: /usr
```

First, I tried the unmodified recipe.  But mesa18 picks python2 and
then fails because it can't find py-mako for python2.

Second, I added PYTHON to the configure args.  (This isn't exactly the
correct setting, but close enough for testing.)

```
'PYTHON=' + spec['python'].prefix.bin.python3
```

Now it finds the right python, but still fails because it doesn't find
the py-mako module.

Third, I tried Adam's patch from #28346 plus PYTHON.  This time it
finds py-mako and works.

----------

Now I see why the patch fixes the configure tests.

Without the patch, `spack-build-env.txt` has these PY* settings.

```
$ grep PY spack-build-env.txt | tr : '\n'
PYTHONHOME=/usr; export PYTHONHOME
PYTHONPATH=/home/krentel/elf/install/linux-rhel8-x86_64/gcc-8.4.1/py-setuptools-59.4.0-g4p4zigdhxwmq6g5hchoyjdw4pymrzvs/lib64/python3.6/site-packages
/home/krentel/elf/install/linux-rhel8-x86_64/gcc-8.4.1/py-markupsafe-2.0.1-vtsnvxhly7i7ewby7erk7gc3s7exahpj/lib64/python3.6/site-packages
/home/krentel/elf/install/linux-rhel8-x86_64/gcc-8.4.1/py-mako-1.1.5-tguj5fykqvzevaidjjspei6ppae463fk/lib64/python3.6/site-packages; export PYTHONPATH
SPACK_PYTHON=/usr/bin/python3; export SPACK_PYTHON
```

With the patch, I get this.

```
$ grep PY spack-build-env.txt | tr : '\n'
PYTHONHOME=/usr; export PYTHONHOME
PYTHONPATH=/home/krentel/mesa/install/linux-rhel8-zen/gcc-8.4.1/py-mako-1.1.5-vkx7pmaxdnf6tvmojajiyafzxdxog66w/lib64/python3.6/site-packages
/home/krentel/mesa/install/linux-rhel8-zen/gcc-8.4.1/py-mako-1.1.5-vkx7pmaxdnf6tvmojajiyafzxdxog66w/lib/python3.6/site-packages
/home/krentel/mesa/install/linux-rhel8-zen/gcc-8.4.1/py-markupsafe-2.0.1-m2rfctaskgq6v5tznokg4xp2w6o4l5yd/lib64/python3.6/site-packages
/home/krentel/mesa/install/linux-rhel8-zen/gcc-8.4.1/py-markupsafe-2.0.1-m2rfctaskgq6v5tznokg4xp2w6o4l5yd/lib/python3.6/site-packages; export PYTHONPATH
SPACK_PYTHON=/usr/bin/python3; export SPACK_PYTHON
```

There are two differences here.  First, there is no python path for
py-setuptools, but that doesn't matter.

What matters is that the patch adds both lib and lib64 paths and the
py-mako files are actually in lib which is missing from the unmodified
version.  As a result, mesa18 configure finds py-mako and the build
succeeds.

So the solution to telling mesa18 the locations of python and py-mako
are: (1) pass PYTHON to the configure args, and (2) add both lib and
lib64 to PYTHONPATH (or possibly, figure out where spack is going to
install them and add that).

Way to go Adam!

Btw, what was the reason for deprecating the older python 3.6 versions?

Glad it worked! I'll submit a PR to set the `PYTHON` env var.

We deprecated Python 3.6 and older in #28003 since these versions are EOL and no longer receiving bug fixes. However, we're reverting this change in #28411 since some users still need Python 2 support and people volunteered to maintain it.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/mesa18/package.py b/var/spack/repos/builtin/packages/mesa18/package.py
--- a/var/spack/repos/builtin/packages/mesa18/package.py
+++ b/var/spack/repos/builtin/packages/mesa18/package.py
@@ -76,6 +76,9 @@ class Mesa18(AutotoolsPackage):
     # Backport Mesa MR#6053 to prevent multiply-defined symbols
     patch('multiple-symbols_hash.patch', when='@:20.1.4%gcc@10:')
 
+    def setup_build_environment(self, env):
+        env.set('PYTHON', self.spec['python'].command.path)
+
     def autoreconf(self, spec, prefix):
         which('autoreconf')('--force',  '--verbose', '--install')
 


Test patch: