martine/ninja — Issue #126: Race condition for rules with multiple targets
Author: PetrWolf
Created: 2011-11-04 18:24:27+00:00
State: open

Please find an example below, which illustrates a situation of incorrect ninja behaviour.

In short, let's assume we have a rule with two outputs. If one of the output files is missing, ninja will regenerate it, along with the second file. However, that second file will not be marked as dirty, and nodes depending on it will not wait for it being re-built.

This confuses ninja (it may crash, or end up with mis-numbered task, eg [4/3]) and creates a race condition (if two tasks accesing the second output file run in parallel).

For the following example, please create 4 files: linux.ninja, test.sh a.in and b.in.

--------------
linux.ninja
--------------
rule cxx
command = cat $in $byproduct > $out

rule cxx_with_byproduct
command = echo Test passed | tee $byproduct > $main_out

rule link
command = cat $in > $out

build a.out: cxx a.in | b.byproduct
byproduct=b.byproduct

build b.out b.byproduct: cxx_with_byproduct b.in
main_out=b.out
byproduct=b.byproduct

build result: link a.out b.out

build all: phony result

----------
test.sh
---------
#!/bin/sh

# Original contens of the byproduct file, which should not be used
echo Test failed > b.byproduct

# Modify a.in to trigger rebuild of a.out
sleep 1
touch a.in

# Delete b.out to trigger rebuild of b.out AND b.byproduct
sleep 1
rm -f b.out
# touch b.in (if input is modified instead of the output, test will pass)

# TEST 1
# run ninja in silent mode - show incorrect order of steps
# sleep 1
# ninja -f linux.ninja -n -v all

# TEST 2
# run ninja (with multiple threads)
sleep 1
ninja -f linux.ninja -j 10 all

# a.out should contain "Test passed",
# which was added to b.byproduct during the build
cat a.out

-----
a.in
-----
Result of the test:

-----
b.in
-----
Not important for the test

----

Then make sure ninja binary exists on your PATH and run test.sh. It should print out

[1/3] cat a.in b.byproduct > a.out
[2/3] echo Test passed | tee b.byproduct > b.out
[3/3] cat a.out b.out > result
[3/3] cat a.out b.out > result
Result of the test:
Test failed

This means that b.byproduct was picked up in task [1/3] before it was modified in [2/3]. Nothing would prevent these two running in parallel.

You can also uncommenti TEST1 in test.sh to see a dry run. This may lead to a crash or misnumbered output (last task is [4/3], cat a.in b.byproduct gets executed twice).

Apparently, this issue seems linked to #118 (Dependent targets on actions that produce multiple outputs).

Short analysis shows, that if an output of a rule is marked as dirty (e.g. because the file is missing), its sibling outputs are left "clean" and thus not expected to be rebuilt.

Can you please confirm whether the above observation is correct and this is indeed a bug?

Thanks
Petr