martine/ninja — Issue #144: Nop bug with restat rules
Author: qhuo
Created: 2011-11-25 21:42:23+00:00
State: open

We just found a nop bug with restat rules. It can be demonstrated with the following example.

$ cat build.ninja
rule generate
depfile = $out.d
command = mv $out $out.tmp && gcc -c -o $out -MMD -MF $out.d $in && rm $out && mv $out.tmp $out
restat = 1

build a.out: generate a.cpp

So the generate rule is used to emulate a code generation tool that happens not to touch its output file. It generates its dep file in the same way as gcc.

We start with an empty a.cpp and a.out.

$ ls -ltr --full-time
total 4
-rw-r--r-- 1 huoq fliclearusers 174 2011-11-25 21:14:27.000000000 +0000 build.ninja
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:18:07.000000000 +0000 a.out
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:18:11.000000000 +0000 a.cpp

$ ninja
[1/1] mv a.out a.out.tmp && gcc -c -o a.out -MMD -MF a.out.d a.cpp && rm a.out && mv a.out.tmp a.out
$ ls -ltr --full-time
total 8
-rw-r--r-- 1 huoq fliclearusers 174 2011-11-25 21:14:27.000000000 +0000 build.ninja
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:18:07.000000000 +0000 a.out
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:18:11.000000000 +0000 a.cpp
-rw-r--r-- 1 huoq fliclearusers 13 2011-11-25 21:20:13.000000000 +0000 a.out.d
$ cat a.out.d
a.out: a.cpp

The first run of ninja is good. It generates a.out.d as we expected, and also keeps a.out unchanged. The next run of ninja is nop, as expected.

$ ninja
ninja: no work to do.

Now to trigger the bug, I will change a.cpp to make it include a.h. I also create a.h, and make it newer than a.cpp.

$ cat a.cpp
#include "a.h"
$ touch a.h
$ ls -ltr --full-time
total 12
-rw-r--r-- 1 huoq fliclearusers 174 2011-11-25 21:14:27.000000000 +0000 build.ninja
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:18:07.000000000 +0000 a.out
-rw-r--r-- 1 huoq fliclearusers 13 2011-11-25 21:20:13.000000000 +0000 a.out.d
-rw-r--r-- 1 huoq fliclearusers 15 2011-11-25 21:23:40.000000000 +0000 a.cpp
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:23:48.000000000 +0000 a.h

Now I run ninja, the build action is run as expected. The only file changed by the build is a.out.d. It has discovered an extra dependency on a.h.

$ ninja
[1/1] mv a.out a.out.tmp && gcc -c -o a.out -MMD -MF a.out.d a.cpp && rm a.out && mv a.out.tmp a.out
$ ls -ltr --full-time
total 12
-rw-r--r-- 1 huoq fliclearusers 174 2011-11-25 21:14:27.000000000 +0000 build.ninja
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:18:07.000000000 +0000 a.out
-rw-r--r-- 1 huoq fliclearusers 15 2011-11-25 21:23:40.000000000 +0000 a.cpp
-rw-r--r-- 1 huoq fliclearusers 0 2011-11-25 21:23:48.000000000 +0000 a.h
-rw-r--r-- 1 huoq fliclearusers 17 2011-11-25 21:25:31.000000000 +0000 a.out.d
$ cat a.out.d
a.out: a.cpp a.h

The next run of ninja is *not* nop. And this is the bug.

$ ninja
[1/1] mv a.out a.out.tmp && gcc -c -o a.out -MMD -MF a.out.d a.cpp && rm a.out && mv a.out.tmp a.out

The next one *is*, although it is one step too late.

$ ninja
ninja: no work to do.

I had a quite look into the ninja source code. It appears that the bug is caused by the design of the restat rule. The restat rule, when detecting a build action does not change its output, records the output file in the .ninja_log file with the newest mtime of all the input of the rule. The problem is ninja cannot always know all the input files used in a build command. In particular, if an #include is added intio the source file, the dependency will be discovered by the compiler, but it is not available to the current ninja run. [Well, I cheated. The test case above was created after I studied the source code]

It might be better to record the current time (when the command is run) into the .ninja_log files as the its logical time. But I don't know the full details of the code so I don't know whether it will work correctly in all cases.