Repository: ruby/rake (Ruby)

Issue:
Multitask transitive dependencies unpredictable.
Not sure if this is a bug or a misunderstanding on my part. I seem to be getting inconsistent race-condition like behavior with the following setup:

Rakefile:
```ruby
multitask :default => [:one, :two] do
end

task :two => :one do
  puts "echo 'should not run?'"
end

task :one do
  raise
end
```

I would expect to never see "should not run?", but I see it intermittently.
```
for i in {1..10}; do rake 2>/dev/null | grep 'should'; done
echo 'should not run?'
echo 'should not run?'
```
multitasks continue their execution even if prerequisite fails
Problem summary: With the following Rakefile rake v12.0.0 continues building the "apps" even though one of its prerequisites (the "sources") fails.

This Rakefile is a dumbed-down version of my application's build system. There are several applications that all depend on a common library. This library in turn depends on several object files, and for each object file there's a single source file. There's a `rule` that compiles those source files into object files.

It's the basic C++ application model: several `.cpp` get turned into `.o`, those `.o` are linked into a `.a`, several applications link against that `.a`.

What happens, though, is that even if turning one of those `.cpp` into an `.o` fails (and therefore the `.a` cannot be built) the linker for the applications is still invoked. That linking step fails, as the library isn't found.

Here's the synthetic Rakefile that reproduces this issue:

```ruby
num_sources   = 20
num_apps      = 8

apps          = (1..num_apps).map { |i| "app#{i}" }
objects       = (1..num_sources).map { |i| "source#{i}.o" }
sources       = (1..num_sources).map { |i| "source#{i}.cpp" }

$stdout_mutex = Mutex.new

def log text
  $stdout_mutex.synchronize {
    puts "#{Time.now} #{text}"
  }
end

task :default => apps

task :clean do
  (apps + objects + ["library"]).each do |name|
    File.unlink(name) if FileTest.exists?(name)
  end

  system "touch #{sources.join(' ')}"
end

compiler = lambda do |*args|
  log "Compiling #{args.first.name}"
  system "sleep 2"

  if args.first.name == "source4.o"
    log "Will throw an exception for source4.o"
    fail
  end
end

rule '.o' => '.cpp', &compiler

file "library" => objects do
  log "Creating library"
  system "touch library"
end

(1..num_apps).each do |i|
  file "app#{i}" => "library" do
    log "Creating app#{i}"
    system "touch app#{i}"
  end
end
```

First, run `rake clean`. This will create the dummy source files `source1.cpp` through `source20.cpp`. Next, run `rake -m`. You'll observe the following output:

```
2017-01-21 12:56:07 +0100 Compiling source1.o
2017-01-21 12:56:09 +0100 Compiling source2.o
2017-01-21 12:56:11 +0100 Compiling source3.o
2017-01-21 12:56:13 +0100 Compiling source4.o
2017-01-21 12:56:15 +0100 Will throw an exception for source4.o
2017-01-21 12:56:15 +0100 Creating app2
rake aborted!

/home/mosu/tmp/Rakefile:34:in `block in <top (required)>'
Tasks: TOP => default => app1 => library => source4.o
2017-01-21 12:56:15 +0100 Creating app4
(See full trace by running task with --trace)
2017-01-21 12:56:15 +0100 Creating app5
```

You can see that Rake aborts building `app1`, but it it still tries to create `app4` and `app5`. Neither of those should have been continued as their prerequisite `library` has failed. Which of those other `appX` targets is continued depends on timing; it's not always 4 and 5.

Just for fun I've tried the same Rakefile with `drake -j8` instead of `rake -m`. `drake` handles this case correctly and doesn't continue building any of the other `appX` targets.



Discussion:
I was able to reproduce.
Working on a fix.
The following PR attempts to fix this issue
 https://github.com/ruby/rake/pull/252

I wonder if this is the same issue I'm experiencing https://github.com/ruby/rake/issues/189.

For what it's worth, in my contrived example the behavior is intermittent.
I don't know, but my log above shows another issue: the items in my example aren't actually executed in parallel. Look at the timestamps of the "Compiling" lines — they're emitted by the `rule`, and those are definitely not in parallel.

Again, drake handles this correctly and executes those `rule`s in parallel. In general I'm pretty disappointed by rake's `multitask` support. drake has had better and more stable support for parallelism for a number of years now.
The following PR attempts to fix this issue
 https://github.com/ruby/rake/pull/252



Patch:
diff --git a/lib/rake/multi_task.rb b/lib/rake/multi_task.rb
index 04c9f3109..3ae363cbe 100644
--- a/lib/rake/multi_task.rb
+++ b/lib/rake/multi_task.rb
@@ -5,46 +5,10 @@ module Rake
   # parallel using Ruby threads.
   #
   class MultiTask < Task
-
-    # Same as invoke, but explicitly pass a call chain to detect
-    # circular dependencies. This is largely copied from Rake::Task
-    # but has been updated such that if multiple tasks depend on this
-    # one in parallel, they will all fail if the first execution of
-    # this task fails.
-    def invoke_with_call_chain(task_args, invocation_chain)
-      new_chain = Rake::InvocationChain.append(self, invocation_chain)
-      @lock.synchronize do
-        begin
-          if @already_invoked
-            if @invocation_exception
-              if application.options.trace
-                application.trace "** Previous invocation of #{name} failed #{format_trace_flags}"
-              end
-              raise @invocation_exception
-            else
-              return
-            end
-          end
-
-          if application.options.trace
-            application.trace "** Invoke #{name} #{format_trace_flags}"
-          end
-          @already_invoked = true
-
-          invoke_prerequisites(task_args, new_chain)
-          execute(task_args) if needed?
-        rescue Exception => ex
-          add_chain_to(ex, new_chain)
-          @invocation_exception = ex
-          raise
-        end
-      end
-    end
-
     private
+
     def invoke_prerequisites(task_args, invocation_chain) # :nodoc:
       invoke_prerequisites_concurrently(task_args, invocation_chain)
     end
   end
-
 end
diff --git a/lib/rake/task.rb b/lib/rake/task.rb
index 256571112..c7e0a1d9e 100644
--- a/lib/rake/task.rb
+++ b/lib/rake/task.rb
@@ -103,6 +103,7 @@ def initialize(task_name, app)
       @scope           = app.current_scope
       @arg_names       = nil
       @locations       = []
+      @invocation_exception = nil
     end
 
     # Enhance a task with prerequisites or actions.  Returns self.
@@ -183,20 +184,39 @@ def invoke(*args)
 
     # Same as invoke, but explicitly pass a call chain to detect
     # circular dependencies.
-    def invoke_with_call_chain(task_args, invocation_chain) # :nodoc:
-      new_chain = InvocationChain.append(self, invocation_chain)
+    #
+    # If multiple tasks depend on this
+    # one in parallel, they will all fail if the first execution of
+    # this task fails.
+    def invoke_with_call_chain(task_args, invocation_chain)
+      new_chain = Rake::InvocationChain.append(self, invocation_chain)
       @lock.synchronize do
-        if application.options.trace
-          application.trace "** Invoke #{name} #{format_trace_flags}"
+        begin
+          if application.options.trace
+            application.trace "** Invoke #{name} #{format_trace_flags}"
+          end
+
+          if @already_invoked
+            if @invocation_exception
+              if application.options.trace
+                application.trace "** Previous invocation of #{name} failed #{format_trace_flags}"
+              end
+              raise @invocation_exception
+            else
+              return
+            end
+          end
+
+          @already_invoked = true
+
+          invoke_prerequisites(task_args, new_chain)
+          execute(task_args) if needed?
+        rescue Exception => ex
+          add_chain_to(ex, new_chain)
+          @invocation_exception = ex
+          raise ex
         end
-        return if @already_invoked
-        @already_invoked = true
-        invoke_prerequisites(task_args, new_chain)
-        execute(task_args) if needed?
       end
-    rescue Exception => ex
-      add_chain_to(ex, new_chain)
-      raise ex
     end
     protected :invoke_with_call_chain
 


Test patch:
diff --git a/test/test_rake_multi_task.rb b/test/test_rake_multi_task.rb
index 8a50c2980..3b767ac0d 100644
--- a/test/test_rake_multi_task.rb
+++ b/test/test_rake_multi_task.rb
@@ -83,4 +83,25 @@ def test_cross_thread_prerequisite_failures
       Rake::Task[:b].invoke
     end
   end
+
+  def test_task_not_executed_if_dependant_task_failed_concurrently
+    multitask :default => [:one, :two]
+
+    task :one do
+      raise
+    end
+
+    task_two_was_executed = false
+    task :two => :one do
+      task_two_was_executed = true
+    end
+
+    begin
+      Rake::Task[:default].invoke
+    rescue RuntimeError
+    ensure
+      sleep 0.5
+      refute task_two_was_executed
+    end
+  end
 end