Repository: conan-io/conan

Issue:
CMake always rerun with Ninja generator
Hi,

I am trying to use conan in our enterprise project and I am having problems when I use it in a CMake project in which I want to use the Ninja generator. This project has many external dependencies, and it has been working normally with any kind of generator (Make, Ninja, MSVC) before trying to use conan.

The first step we tried in our migration is to handle the boost dependency. When we use the Make generator everything works like a charm. However when I try to use the Ninja generator, every time I run the ninja command, CMake is rerunning the configuration many times.

I tried to run Ninja with "-d explain" to determine the origin of the problem. For some reason many of the cmake files are considered as _dirty_. 

```
# Configure project
mkdir build
conan install .. && cmake -GNinja ..

# Run ninja with debugging capabilities
luis@p4dDesktop:~/projects/pix4d/master-conan/pix4dmapper/build$ ninja -d explain
ninja explain: output /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake of phony edge with no inputs doesn't exist
ninja explain: /home/luis/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6/FindBoost.cmake is dirty
ninja explain: /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake is dirty
...
ninja explain: /home/luis/projects/Pix4DMapper-Master/master-conan/pix4dmapper/src/apps/CMakeLists.txt is dirty
# A bunch of other project cmake files
...
ninja explain: /usr/local/share/cmake-3.6/Modules/AutogenInfo.cmake.in is dirty
ninja explain: /usr/local/share/cmake-3.6/Modules/CMakeCCompiler.cmake.in is dirty
ninja explain: /usr/local/share/cmake-3.6/Modules/CMakeCCompilerABI.c is dirty
# Many other cmake files
...
ninja explain: CMakeCache.txt is dirty
ninja explain: CMakeFiles/3.6.2/CMakeCCompiler.cmake is dirty
ninja explain: CMakeFiles/3.6.2/CMakeCXXCompiler.cmake is dirty
ninja explain: CMakeFiles/3.6.2/CMakeSystem.cmake is dirty
ninja explain: CMakeFiles/feature_tests.c is dirty
ninja explain: CMakeFiles/feature_tests.cxx is dirty
ninja explain: conanbuildinfo.cmake is dirty
[0/1] Re-running CMake...

```

Note that when I remove the following lines from my CMakeLists.txt file, the ninja generator starts to work again:

```
#include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
#conan_basic_setup()
```

Any of you has experienced similar issues?



Discussion:
Could be an issue with the conan file's timestamps, but we need to debug it, could be other thing. No similar problems detected previously. Thanks for reporting.

I have been trying for a while, Ubuntu 14, with dependencies to Boost & Poco. 
First, everything worked fine, I thought that it was because I was not using the `find_package(Boost` feature. I added it, and still the same, it build fast, not "dirty" files problem.

So unable to reproduce it yet :(

As info, also using CMake 3.6 (from the conan `cmake_installer/0.1@lasote/stable` package). Maybe cmake issue?

Update: Tried with CMake 3.1, the same.

Using Ninja version 1.3.4
Indeed it could be an issue with the dates of the files, https://github.com/ninja-build/ninja/issues/1120, but don't know why it happens to you and not me.

I will try later with different versions of ninja and CMake. Right now I am using the latest version of both (1.7.1 and 3.6.2 respectively)

Interesting! I tried with the ninja package provided by Ubuntu 16.04 (version 1.5.1) and the situation has improved. Instead of rerunning CMake forever now it only do it once, and then the compilation starts. However, each time that I run ninja, is doing re-running cmake again (only once). 

Then I tried with the latest version of ninja (from their github repository) and I fell into the same problem (re-running cmake forever).

What is strange for me is that this situation only happens if I include the conan file. 

@memsharded I tried with the same ninja version than you (1.3.4) and It is also re-running once cmake.

The strange thing is that I also tried to reproduce the same issue with a hello-world project and it is not happening :weary: 

I will let you know if I discover the problem. 

@piponazo Cannot reproduce the error either. :weary: I've uploaded a new conan installer recipe with virtualenv (ninja_installer/0.1@lasote/testing) to ease trying several versions but it works just fine.
Could you paste the failing conanfile to see the dependency tree?

Sure. I am just depending on my personal boost package. This is the conanfile:

```
[requires]
Boost/1.60.0@piponazo/testing

[generators]
cmake

[options]
Boost:shared=True
Boost:fPIC=True
Boost:with_date_time=True
Boost:with_filesystem=True
Boost:with_iostreams=True
Boost:with_program_options=True
Boost:with_serialization=True
Boost:with_system=True
Boost:with_thread=True
```

As I told before I tried to reproduce the issue in the same machine (office desktop) with a smaller project with no luck. However, yesterday I tried at home with other small project and I could reproduce it. I will try again in that machine this evening ;)

When you say with a "smaller project", you mean not depending on your personal boost package?  Thanks!

No, I am sorry for the confusion. Out enterprise project has many software dependencies, and right now I only removed **boost** from our previous "package manager" and I started to use the **boost** package provided by conan.

Since this project is so big (many other dependencies, a lot of complex cmake stuff, and so on) I wonder if this issue was triggered by something else. I tried to reproduce the issue with a simpler hello-world project making use of **boost**. So in that smaller project, I use the same conanfile, then I include the cmake file generated with conan install and I included the boost folders and libraries in the same way that I do in the big project.

I could not spend much more time on this thing at work, but I will try to investigate it in my spare time :)

I was able to reproduce the issue in my personal laptop.
- This is the project in which I can easily reproduce the issue: https://github.com/piponazo/conanIssueNinja.
- I am running conan on this computer using pythonenv. Conan version 0.13.2.
- System: Ubuntu 16.04
- Gcc 5.4.0-6ubuntu1~16.04.2 (from distro)
- Ninja 1.7.1 (binaries from the web)
- CMake 3.6.2 Installed from the sources

Here you can find some "relevant?" system details:

```
H/W path        Device     Class          Description
=====================================================
                           system         W740SU (Not Applicable)
/0                         bus            W740SU
/0/0                       memory         64KiB BIOS
/0/14                      processor      Intel(R) Core(TM) i7-4760HQ CPU @ 2.10GHz
/0/14/15                   memory         1MiB L2 cache
/0/14/16                   memory         256KiB L1 cache
/0/14/17                   memory         6MiB L3 cache
/0/18                      memory         16GiB System Memory
/0/18/0                    memory         8GiB SODIMM DDR3 Synchronous 1600 MHz (0,6 ns)
...
/0/2/0.0.0      /dev/sdb   disk           240GB Crucial_CT240M50
/0/2/0.0.0/1    /dev/sdb1  volume         125GiB Windows NTFS volume
/0/2/0.0.0/3    /dev/sdb3  volume         97GiB Extended partition
/0/2/0.0.0/3/5  /dev/sdb5  volume         23GiB Linux filesystem partition
/0/2/0.0.0/3/6  /dev/sdb6  volume         74GiB Linux filesystem partition

# I am running the experiments on the /dev/sdb6 partition (SSD Disk)
```

This is exactly the sequence of commands that I run to reproduce the issue:

```
# Open a terminal
1) source env/env1/bin/activate # enter in the python environment where I installed conan
2) cd conanExample
3) mkdir build && cd build
4) conan install .. && cmake -GNinja ..
5) ninja
```

Conan config file:

```
[storage]
path: ~/.conan/data

[proxies]

[settings_defaults]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=5.4
```

Resulting packages in the .conan/data/ directory: **Boost  bzip2  zlib**.

If you need any other detail, please, do not hesitate to ask me. 

Yes, please, exact versions. 16.04 comes with gcc 5.4, but conan doesn't have binaries for some deps as bzip2. So:
gcc:
cmake: 3.6? Installed from?
Ninja?

Thanks!

Did a test with your repo and:
- conan 0.13.3 (from pip)
- Ubuntu 16.04 (VM)
- gcc 4.9
- cmake 3.6 (from cmake_installer conan package) and 3.5 (from distro)
- Ninja 1.5.1 (from distro).

Still not able to reproduce :(

I just updated the previous comment to include all the missing information you asked for ;)

As a side observation, you might want to change your default libstdc++ in `conan.conf` to libstdc++11, which would be the default in Ubuntu 16 with gcc 5.4

@piponazo Could you please list with details the contents of: `/home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/` ?

I am especially interested in FindZLib.cmake, could you please run `stat` on it? Thanks!

aha! I think that you have caught the problem!

```
luis@luis-W740SU:~$ ls /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/ -lha
total 28K
drwxrwxr-x 4 luis luis 4,0K oct 11 22:39 .
drwxrwxr-x 3 luis luis 4,0K oct 11 22:39 ..
-rw-rw-r-- 1 luis luis  314 oct 11 22:39 conaninfo.txt
-rw-rw-r-- 1 luis luis 1,6K oct 11 22:39 conanmanifest.txt
-rw-rw-r-- 1 luis luis 2,1K ene  1  1970 FindZLIB.cmake
drwxrwxr-x 2 luis luis 4,0K oct 11 22:39 include
drwxrwxr-x 2 luis luis 4,0K oct 11 22:39 lib

luis@luis-W740SU:~$ stat /home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake 
  File: '/home/luis/.conan/data/zlib/1.2.8/lasote/stable/package/21ace02f4960dd0c1d50bd3abe1537054de08157/FindZLIB.cmake'
  Size: 2093        Blocks: 8          IO Block: 4096   regular file
Device: 816h/2070d  Inode: 787762      Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/    luis)   Gid: ( 1000/    luis)
Access: 1970-01-01 01:00:00.000000000 +0100
Modify: 1970-01-01 01:00:00.000000000 +0100
Change: 2016-10-11 22:39:21.829907608 +0200
 Birth: -
```

Does this means that the FindZLIB.cmake file in the recipe has a bad date?

By the way, yes I know about the libstdc++11 thing (I read carefully the conan.io documentation hehe). I am using that in my desktop machine in the office but the file I sent you was generated in my personal laptop were I did not spend time configuring this file ;). Anyway thanks!

Note that the file in the source folder already has that date:

```
luis@luis-W740SU:~$ ls /home/luis/.conan/data/zlib/1.2.8/lasote/stable/source/ -lha
total 44K
drwxrwxr-x  3 luis luis 4,0K oct 11 22:39 .
drwxrwxr-x  6 luis luis 4,0K oct 11 22:39 ..
-rw-rw-r--  1 luis luis  190 ene  1  1970 CMakeLists.txt
-rw-rw-r--  1 luis luis 3,2K oct 11 22:39 conan_export.tgz
-rw-rw-r--  1 luis luis 5,1K oct 11 22:39 conanfile.py
-rw-rw-r--  1 luis luis 4,8K oct 11 22:39 conanfile.pyc
-rw-rw-r--  1 luis luis  155 oct 11 22:39 conanmanifest.txt
-rw-rw-r--  1 luis luis 2,1K ene  1  1970 FindZLIB.cmake
drwxrwxr-x 14 luis luis 4,0K oct 11 22:39 zlib-1.2.8
```

Yes, it is possible. The .tgz used to transfer files doesn't use file dates, so always compress to the same checksum. What it is weird is why it happens in your system and not in ours. Also, I have been trying to manually change the `mtime` date to the failing one both for Ninja 1.5 and 1.7, without success.

Also, still don't know why other files such as those of cmake are marked as dirty, definitely conan has nothing to do there, and it doesn't modify or touch them in any way.

In any case, a small step forward, lets keep working on it, and we will find that damn bug! :)

At home I have two computers (desktop + laptop). I could reproduce the issue in both of them. In the laptop I run Ubuntu 16.04 and in the desktop Xubuntu 16.04. Both pcs have SSDs. At first I was suspecting about this and the fact that I use the option **noatime** when mounting the /home partition. So I tried to remove that option, remove all the packages from the conan/data folder and repeat the process. The result is the same.

I also tried to move the conan/data folder to partition mounted in a non-SDD hard-drive and repeat the previous steps. No luck here neither.

I have been trying some things and there is something that I found very weird. We have some files in the source folders of conan having EPOCH in their dates. We don't know why but that's fine for the moment. 

However if I take one of those files and I copy it to the tmp folder this is what happen:

```
# Checking that the original file is in EPOCH
ls -lh ~/.conan/data/Boost/1.60.0/lasote/stable/source/FindBoost.cmake 
-rw-r--r-- 1 luis luis 82K ene  1  1970 /home/luis/.conan/data/Boost/1.60.0/lasote/stable/source/FindBoost.cmake

# I copy that file to other place
cp ~/.conan/data/Boost/1.60.0/lasote/stable/source/FindBoost.cmake /tmp/

# I check its time in the new place
ls /tmp/FindBoost.cmake  -lh
-rw-r--r-- 1 luis luis 82K oct 15 18:37 /tmp/FindBoost.cmake
```

Why is not happening the same when conan is copying that file to the **package** folders? Is it a limitation in python? Is there any option to configure this?

When I compile the boost package I can find these lines in the log:

```
Boost/1.60.0@lasote/stable: Generating the package
Boost/1.60.0@lasote/stable: Package folder /home/luis/.conan/data/Boost/1.60.0/lasote/stable/package/8b111290ce72aaeabe5c7ccfbf008ab6905362aa
Boost/1.60.0@lasote/stable package(): Copied 136 '.h' files
Boost/1.60.0@lasote/stable package(): Copied 3 '.pl' files: gen_function_N.pl, gen_maybe_include.pl, gen_signal_N.pl
Boost/1.60.0@lasote/stable package(): Copied 3 '.m4' files: for.m4, pool_construct_simple.m4, pool_construct.m4
Boost/1.60.0@lasote/stable package(): Copied 1 '.cmake' files: FindBoost.cmake
...
```

But strangely, the FindBoost.cmake file still have the EPOCH time in the package folder.

That is a very good investigation into the causes. I will also try a few things with your ideas, thanks!

I have done a few further steps, I'd say we are very close:

I have tracked down the issues in this order:
- I was confused because I couldn't reproduce the issue by hand modifying the file dates, with the `touch` command, the issue was that I was changing only the time, but not the date.
- when I changed the date & time to 0, everything seems to work fine. It results that **ninja** uses epoch+1 to indicate no file.
- Files are tgz in conan with mtime=0, so they do not generate a different hash
- When files are decompressed, they are generated in disk by python with epoch+1, as default. Bad luck.
- Conan is actually doing a "touch" over the decompressed files. Internally it is an `os.utime(file, None)` which should use the current time. This work on my system, but I bet it is failing on yours, don't know why yet
- I think there could be a bug in the `touch()` function, opening the file before running the `os.utime` which I think it is unnecessary: I have just submitted a PR to remove it: https://github.com/conan-io/conan/pull/566. If you want to do a quick check, you could do the same change in your local installation, just search for the `conans/util/files.py` file and change it as in the PR.
- Another simple test you could run, is to run the python command `os.utime(filename, None)` over a test file and check if it actually changes the mtime to the current one.

Lets kill this bug!! :D :D

I just tried the two things you proposed.

In the first case, I deleted my conan installation (I installed using the linux installer) and installed again using pythonenv + pip. Then I modified the file mentioned in #566 and I tried to recompile the packages again. Sadly, I am still falling in the same situation. The file _FindBoost.cmake_ is still en epoch in the package folder. 

However, in the second test, I did this:

```
luis@p4dDesktop:~/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6$ ls -lh
total 996K
-rw-rw-r-- 1 luis luis 1.1K Oct 17 09:08 conaninfo.txt
-rw-rw-r-- 1 luis luis 900K Oct 17 09:08 conanmanifest.txt
-rw-rw-r-- 1 luis luis  82K Jan  1  1970 FindBoost.cmake
drwxrwxr-x 3 luis luis 4.0K Oct 17 09:08 include
drwxrwxr-x 2 luis luis 4.0K Oct 17 09:08 lib
luis@p4dDesktop:~/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.utime("FindBoost.cmake", None)
>>> 
luis@p4dDesktop:~/.conan/data/Boost/1.60.0/piponazo/testing/package/ed2b408ce34ce36caef16f74181d3bc588210ba6$ ls -lh
total 996K
-rw-rw-r-- 1 luis luis 1.1K Oct 17 09:08 conaninfo.txt
-rw-rw-r-- 1 luis luis 900K Oct 17 09:08 conanmanifest.txt
-rw-rw-r-- 1 luis luis  82K Oct 17 09:13 FindBoost.cmake
drwxrwxr-x 3 luis luis 4.0K Oct 17 09:08 include
drwxrwxr-x 2 luis luis 4.0K Oct 17 09:08 lib
```

As you can see, it's updating the file correctly. 

Please, with the modified installation try to remove `conan remove "*" -f` your local cache, then install it again, then rebuild. 

The `touch` is done at `untargz` time, that is done at install if not already installed.

Another test you could do if that doesn't work is to insert a `os.system("stat " + fname)` before and after the `os.utime`, to check if it is changing the file mtime or not.

I tried both things you mentioned and I ended up in the same situation. The FindBoost.cmake package in the package folder is still in epoch. That's very weird. Are you sure that you are calling that **touch()** function when the package is installed after its compilation? 

What was the output of putting such `os.system` before and after the `os.time`, did they execute showing the same output, or maybe they didn't execute? If you want to share screen, we may try to do a pair-programming debug session.

I have another idea:

``` python
for a, b, c in os.walk("afolder"):
    print a, b, c
```

What does this prints if "afolder" does not exist? NOTHING. That could be the case for:

``` python
def get_package(self, package_reference, dest_folder, remote):
        """
        Read the conans package from remotes
        Will iterate the remotes to find the conans unless remote was specified

        returns (dict relative_filepath:abs_path , remote_name)"""
        zipped_files = self._call_remote(remote, "get_package", package_reference, dest_folder)
        files = unzip_and_get_files(zipped_files, dest_folder, PACKAGE_TGZ_NAME)
        # Issue #214 https://github.com/conan-io/conan/issues/214
        for dirname, _, files in os.walk(dest_folder):
            for fname in files:
                touch(os.path.join(dirname, fname))

        return files
```

It could be a case of fast execution, that the dest_folder is still not created, even if it has been commanded to create previously, so the iteration of `os.walk` does nothing.

I know it is very difficult, as it is running a extract of the tgz before. What it is clear, is that it is important to monitor that `touch` is indeed being called.


Resolving patch:
diff --git a/conans/client/remote_manager.py b/conans/client/remote_manager.py
--- a/conans/client/remote_manager.py
+++ b/conans/client/remote_manager.py
@@ -97,6 +97,9 @@ def get_recipe(self, conan_reference, dest_folder, remote):
         files = unzip_and_get_files(zipped_files, dest_folder, EXPORT_TGZ_NAME)
         # Make sure that the source dir is deleted
         rmdir(self._client_cache.source(conan_reference), True)
+        for dirname, _, filenames in os.walk(dest_folder):
+            for fname in filenames:
+                touch(os.path.join(dirname, fname))
 #       TODO: Download only the CONANFILE file and only download the rest of files
 #       in install if needed (not found remote package)
         return files
@@ -110,8 +113,8 @@ def get_package(self, package_reference, dest_folder, remote):
         zipped_files = self._call_remote(remote, "get_package", package_reference, dest_folder)
         files = unzip_and_get_files(zipped_files, dest_folder, PACKAGE_TGZ_NAME)
         # Issue #214 https://github.com/conan-io/conan/issues/214
-        for dirname, _, files in os.walk(dest_folder):
-            for fname in files:
+        for dirname, _, filenames in os.walk(dest_folder):
+            for fname in filenames:
                 touch(os.path.join(dirname, fname))
 
         return files


Test patch: