Repository: mesonbuild/meson

Issue:
ninja/cmake issue when importing large subproject
**Describe the bug**
I am using meson to compile a complicated project. Part of this project involves sharing C++ source between my x86 spec PC and an ESP32 micro-controller. When I attempt to connect the ESP32 SDK's cmake project to my meson project as a subproject, ninja stops with error:

`ERROR: Multiple producers for Ninja target "subprojects/esp32/55ac3a2@@cm_mbedcrypto@sta/bignum.c.o". Please rename your targets.`

**To Reproduce**
You'll need the ESP-IDF toolchain, setup guide here:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/

I've created an example repo with a trivial ESP32 main implementation as an example here:
https://github.com/Michael-Densberger/meson_esp32_issue_example

**Expected behavior**
I expected that the cmake subproject would build and I could use custom meson targets to interact with the idf.py wrapper that is used by the ESP-IDF toolchain to manage the micro-controller. Indeed, calling `idf.py flash monitor` from the subproject/esp32 dir results in a compiling and running binary on my ESP32 micro-controller. 

To clarify:
1. using cmake through the idf.py wrapper:
    works with no issues.
2. using cmake through meson results in:
    `ERROR: Multiple producers for Ninja target "subprojects/esp32/55ac3a2@@cm_mbedcrypto@sta/bignum.c.o". Please rename your targets.`

**system parameters**
* Native build
* An as of this morning up-to-date Arch Linux
* Python 3.8.2
* Meson 0.54.0
* Ninja 1.10.0




Discussion:
Could you please provide a list of AUR packages to install for reproducing this (I am also on Arch). Also please provide the `build/meson-logs/meson-log.txt`, the `build/subprojects/<your sp>/{meson.build,fileAPI.json}` files and `build/subprojects/<your sp>/__CMake_build/cmake_trace.txt`. With pastebin, or soemthing else of course.
Note there is a AUR versioning issue here, the AUR package is out of date. I am getting a new, different error with this package.

Setup:
I ran in this order:
`mkdir /opt/ESP-IDF`
`cd /opt/ESP-IDF`
`git clone https://aur.archlinux.org/esp-idf.git .`
`makepkg -si`

We get into the weeds a bit here. Long story short:
Arch installs python virtualenv with a version greater than 20 at the moment.
In version 20, virtualenv removed the `'--no-site-packages'` switch.
The install script that must be run post pacman install supplies this switch.
We need to remove the switch from the install script.

I did this with the following:
`sudo nano /opt/esp-idf/tools/idf_tools.py`
`ctrl+w`
search for `'--no-site-packages'`
line 1155 was: 

        `subprocess.check_call([sys.executable, '-m', 'virtualenv', '--no-site-packages', idf_python_env_path], sys.stdout, stderr=sys.stderr)`

Now:

        `subprocess.check_call([sys.executable, '-m', 'virtualenv', idf_python_env_path], sys.stdout, stderr=sys.stderr)`

Next I ran:
`/opt/esp-idf/install.sh`
`. /opt/esp-idf/export.sh`

We're all set up and ready now, to build the example:

`mkdir /opt/meson_esp_example`
`cd /opt/meson_esp_example`
`git clone git@github.com:Michael-Densberger/meson_esp32_issue_example.git .`
`cd subprojects/esp32/`
`idf.py -p /dev/ttyUSB0 flash monitor` - this works and produces the "hello world!" output expected.
`ctrl+]` - end the monitoring session
`cd ../..`
`meson . build/` - result: `meson.build:4:0: ERROR: Unknown target type "INTERFACE_LIBRARY"`
Note there is a warning about: `WARNING: CMake: path /opt/meson_esp_example/build/subprojects/esp32/__CMake_build/dummy_main_src.c.rule does not exist. Ignoring. This can lead to build errors` 

I am still troubleshooting this new error, any advice would be helpful.

meson-log.txt: https://pastebin.com/5PnADxiu

The other files are not present, I assume because this new error is earlier in the meson process.
Ah, the `. /opt/esp-idf/export.sh` activates a virtual environment with an older version of meson installed. I'll separately inform the maintainers of the AUR package to update, and I'll inform the maintainers of the ESP-IDF toolchain about this issue. 

Now we are getting a meson assertion error, meson-log.txt pastebin here: https://pastebin.com/wC0MDTF1

fileAPI.json and cmake_trace.txt are present but huge (for text files), do you have a preferred file transfer service?

edit, the paste did not capture the assertion error, here it is: https://pastebin.com/8wxGKgKH

I am going to continue troubleshooting this and keep you updated, I don't know if we're seeing a configuration issue or a bug here to be honest.
Okay, reproducing the original problem using the AUR package. Update below.

First, we need to step away from the export.sh script:
  1. I modified my path to include the following directories:
    `/home/michael/.espressif/tools/xtensa-esp32-elf/1.22.0-80-g6c4433a-5.2.0/xtensa-esp32-elf/bin/`
    `/opt/esp-idf/tools`
  2. I created the environment variable IDF_PATH
    `export IDF_PATH=/opt/esp-idf/`

Now we can interact with the ESP-IDF toolchain without calling `export.sh`

Second, we run `meson . build` to produce the first issue.
  1. In meson src file `.../mesonbuild/cmake/interpreter.py`
  Line 604|605:
    `assert(target.current_bin_dir is not None)`
    `assert(target.current_src_dir is not None)`
  The input `CMakeGeneratorTarget` does not have these members set. 
  In a kludge, I just set them 
    `target.current_bin_dir = "."`
    `target.current_src_dir = "."`

Third, the original problem comes back if we run: `meson . build`
  1. We eventually get to a problem resembling the original:
  `ERROR: Multiple producers for Ninja target "subprojects/esp32/55ac3a2@@cm_idf_component_lwip@sta/sntp.c.o". Please rename your targets.`

I see two main issues here: the kludge to skip past the first error and the original issue.

I am going to troubleshoot further on the kludge and figure out if I can adapt the input CMakeLists.txt defined in my example repo to define these behaviors. 

For the second issue, I think that is more in your court. Let me know if the fileAPI.json, etc. is relevant now and I'll upload it.



The assertion occurs because meson assumes that there is a `project()` call pretty much at the start of the `CMakeLists.txt` (especially before any `add_custom_command`, etc.). The following patch to your project fixes at least the first issue for me:

```diff
diff --git a/subprojects/esp32/CMakeLists.txt b/subprojects/esp32/CMakeLists.txt
index 16be5c1..ad6f7b6 100644
--- a/subprojects/esp32/CMakeLists.txt
+++ b/subprojects/esp32/CMakeLists.txt
@@ -1,3 +1,3 @@
 cmake_minimum_required(VERSION 3.5)
-include($ENV{IDF_PATH}/tools/cmake/project.cmake)
-project(esp32)
\ No newline at end of file
+project(esp32)
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
\ No newline at end of file
```
Also, the virtual environment does not contain a meson package for me (fresh install obviously). Maybe you ran `pip install meson` at some point with the export script activated?
I ran some more tests and was unable to reproduce the errors (the `INTERFACE_LIBRARY` error was due to using the old meson version). Could you please check if reordering the `project()` call also works for your original project?

To explain why the project call is important: The `project()` function allows arbitrary [code injection](https://cmake.org/cmake/help/latest/command/project.html#code-injection) via a command-line flag. We use this to keep track of `CMAKE_CURRENT_BINARY_DIR` and `CMAKE_CURRENT_SOURCE_DIR` for custom commands via our [`preload.cmake`](https://github.com/mesonbuild/meson/blob/master/mesonbuild/cmake/data/preload.cmake) script. The main disadvantage is that everything before *any* project call is not affected and thus it is impossible to track those variables.
Warning, complicated version-itis ahead:

Both versions (the Arch AUR package and the latest from the github repo) produce differing numbers of targets depending on where the `project(...)` line is in the top level CMakeLists.txt

1. Using the AUR package:
  * Top level defined as:
    `cmake_minimum_required(VERSION 3.5)`
    `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
    `project(esp32)`

    result is an assertion error (as expected): https://pastebin.com/4XmkNyqJ

  * Top level defined as:
    `cmake_minimum_required(VERSION 3.5)`
    `project(esp32)`
    `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`

    result is meson and ninja claiming success with 30 targets(this seems wrong): https://pastebin.com/40qpWr0y

2. Using the latest version (which has not yet encountered the assertion error problem):
  * Top level defined as:
    `cmake_minimum_required(VERSION 3.5)`
    `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
    `project(esp32)`

    result is the original error but 121 targets (this seems right): https://pastebin.com/9H3AuJz4

  * Top level defined as:
    `cmake_minimum_required(VERSION 3.5)`
    `project(esp32)`
    `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`

    result is meson and ninja claiming success with 0 targets(this is almost certainly wrong): https://pastebin.com/40qpWr0y   https://pastebin.com/NTs5iCkt

Number 2 bullet 1 is my prime candidate. In meson, a `message(sub_proj.target_list())` produces expected output, the various libraries and custom targets needed to build an ESP32 app. It is unfortunately also producing the `ERROR: Multiple producers for Ninja target "subprojects/esp32/55ac3a2@@cm_mbedcrypto@sta/bignum.c.o". Please rename your targets.` error.

P.S.

My kludge to hack a path onto the `CMakeGeneratorTarget` with the AUR package version also produces reasonable target output:
  * Top level defined as:
    `cmake_minimum_required(VERSION 3.5)`
    `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
    `project(esp32)`
   
  * interpreter.py hack
        `# assert(target.current_bin_dir is not None)`
        `# assert(target.current_src_dir is not None)`
        `target.current_bin_dir = "."`
        `target.current_src_dir = "."`

   result is the same `Multiple producers...` issue but a target count of 250 and a reasonable target list. The differing target count is expected in my opinion as the cmake project is obviously different between the versions. Full log: https://pastebin.com/P7EuUkzt

Something is wrong here I think, please let me know if I can gather anything else to make troubleshooting easier.

Could you test if there is a difference between the two different `project()` placements when building with CMake only, so we can test if CMake actually does something else if a `project()` is before the `include()`?
1. AUR version
  * Top level defined as:
  `cmake_minimum_required(VERSION 3.5)`
  `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
  `project(esp32)`
    1. Issued command: `cmake -G Ninja -S . -B ./build`
    2. Results in no errors: https://pastebin.com/r3c48XVS
    3. Issued command: `ninja -C build/`
    4. Results in no errors: https://pastebin.com/dDPgeUjM

  * Top level defined as:
  `cmake_minimum_required(VERSION 3.5)`
  `project(esp32)`
  `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
    1. Issued command: `cmake -G Ninja -S . -B ./build`
    2. Results in no errors but minimal output: https://pastebin.com/xreXmbCy
    3. Issued command: `ninja -C build/`
    4. Results `ninja: no work to do.` even on a new build directory.

2. Latest version
  * Top level defined as:
  `cmake_minimum_required(VERSION 3.5)`
  `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
  `project(esp32)`
    1. Issued command: `cmake -G Ninja -S . -B ./build`
    2. Results in no errors: https://pastebin.com/KELgsRgE
    3. Issued command: `ninja -C build/`
    4. Results in no errors: https://pastebin.com/NcCiHatk

  * Top level defined as:
  `cmake_minimum_required(VERSION 3.5)`
  `project(esp32)`
  `include($ENV{IDF_PATH}/tools/cmake/project.cmake)`
    
    Results are identical to the AUR version, minimal cmake output and ninja reporting no work to do even on a new build directory.

My takeaways:

1. The changes made to the latest version solve the meson current_bin_dir etc. issues. 
2. `project(...)` before the include results in cmake not seeing the larger cmake project.





I also think I figured out the cause of the `bignum.c.o` error:

In the target ``, there are two files that are named `bignum.c`:
- /opt/esp-idf/components/wpa_supplicant/src/crypto/bignum.c
- /opt/esp-idf/components/wpa_supplicant/src/wpa2/tls/bignum.c

Because these files are outside the root project dir, meson seems to be having trouble generating unique names for the files, resulting in a name collision.

Resolving patch:
diff --git a/mesonbuild/backend/backends.py b/mesonbuild/backend/backends.py
--- a/mesonbuild/backend/backends.py
+++ b/mesonbuild/backend/backends.py
@@ -468,8 +468,8 @@ def object_filename_from_source(self, target, source):
             source = 'meson-generated_' + os.path.relpath(rel_src, targetdir)
         else:
             if os.path.isabs(rel_src):
-                # Not from the source directory; hopefully this doesn't conflict with user's source files.
-                source = os.path.basename(rel_src)
+                # Use the absolute path directly to avoid file name conflicts
+                source = rel_src
             else:
                 source = os.path.relpath(os.path.join(build_dir, rel_src),
                                          os.path.join(self.environment.get_source_dir(), target.get_subdir()))


Test patch: