Repository: SCons/scons

Issue:
scons -j num causes dependency cycle, no cycle found for node
This issue was originally created at: 2015-12-29 04:13:46.
This issue was reported by: `anticimex`.
anticimex said at 2015-12-29 04:13:46
>Referring to the scons-users thread at 
>https://pairlist4.pair.net/pipermail/scons-users/2015-November/004224.html , 
>there is a bug with parallel builds, provoked by all of variant_dir/SideEffect/parallel.
> 
>The bug:
```
$ scons -j 8
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated VariantDir targets: build
make_file(["build/output"], ["test.cpp"])
scons: done building targets.
 
scons: *** Found dependency cycle(s):
  Internal Error: no cycle found for node build (<SCons.Node.FS.Dir object at 
0x25eb210>) in state pending
 
File "/usr/lib/scons/SCons/Taskmaster.py", line 1026, in cleanup
```

> 
>The test code.
>SConstruct:
>==================
```
env = Environment()
 
def make_file(target, source, env):
    with open(str(target[0]), 'w') as f:
        f.write('gobldygook')
    with open(str(target[0]) + '_side_effect', 'w') as side_effect:
        side_effect.write('anything')
 
env.Append(BUILDERS =
    {'make_file': Builder(action=Action(make_file))})
 
SConscript('SConscript',
     variant_dir = 'build',
     exports={'env':env},
     duplicate=0)
```
>==================
> 
>SConscript:
>==================
```
Import('env')
 
primary = env.make_file('output', 'test.cpp')
this_causes_problems = env.SideEffect('output_side_effect', 'output')
```
>==================
> 
>test.cpp:
>==================
```
#include <iostream.h>
 
main()
{
    cout << "Hello World!";
    return 0;
}
```
>==================
> 
> 
>In our case, we're trying to build a project with ~200 SConscript files that 
>fails like this bug, when doing parallel build, and serial build just takes so 
>long (>45m).
>Our dev cycles would improve severely if we were able to properly do parallel 
>builds in the project.


Votes for this issue: 2.



Discussion:
This looks like Issue #3013 and has a fix in a pull request that's been pending on bitbucket for over two years: https://bitbucket.org/scons/scons/pull-requests/303/fix-handling-of-side-effects-in-taskmaster/diff

Resolving patch:
diff --git a/src/engine/SCons/Taskmaster.py b/src/engine/SCons/Taskmaster.py
--- a/src/engine/SCons/Taskmaster.py
+++ b/src/engine/SCons/Taskmaster.py
@@ -470,14 +470,23 @@ def postprocess(self):
                 pending_children.discard(t)
             for p in t.waiting_parents:
                 parents[p] = parents.get(p, 0) + 1
+            t.waiting_parents = set()
 
         for t in targets:
             if t.side_effects is not None:
                 for s in t.side_effects:
                     if s.get_state() == NODE_EXECUTING:
                         s.set_state(NODE_NO_STATE)
+                        
+                    # The side-effects may have been transferred to
+                    # NODE_NO_STATE by executed_with{,out}_callbacks, but was
+                    # not taken out of the waiting parents/pending children
+                    # data structures. Check for that now.
+                    if s.get_state() == NODE_NO_STATE and s.waiting_parents:
+                        pending_children.discard(s)
                         for p in s.waiting_parents:
                             parents[p] = parents.get(p, 0) + 1
+                        s.waiting_parents = set()
                     for p in s.waiting_s_e:
                         if p.ref_count == 0:
                             self.tm.candidates.append(p)


Test patch:
diff --git a/test/SideEffect/Issues/3013/files/SConscript b/test/SideEffect/Issues/3013/files/SConscript
new file mode 100644
--- /dev/null
+++ b/test/SideEffect/Issues/3013/files/SConscript
@@ -0,0 +1,5 @@
+Import('env')
+
+primary = env.make_file('output', 'test.cpp')
+this_causes_problems = env.SideEffect('output_side_effect', 'output')
+
diff --git a/test/SideEffect/Issues/3013/files/SConstruct b/test/SideEffect/Issues/3013/files/SConstruct
new file mode 100644
--- /dev/null
+++ b/test/SideEffect/Issues/3013/files/SConstruct
@@ -0,0 +1,21 @@
+env = Environment()
+
+def make_file(target, source, env):
+    with open(str(target[0]), 'w') as f:
+        f.write('gobldygook')
+    with open(str(target[0]) + '_side_effect', 'w') as side_effect:
+        side_effect.write('anything')
+
+env.Append(
+    BUILDERS={'make_file': Builder(action=Action(make_file))}
+)
+
+env.objdir = 'build'
+
+SConscript(
+    'SConscript',
+    variant_dir=env.objdir,
+    exports={'env':env},
+    duplicate=0
+)
+
diff --git a/test/SideEffect/Issues/3013/files/test.cpp b/test/SideEffect/Issues/3013/files/test.cpp
new file mode 100644
--- /dev/null
+++ b/test/SideEffect/Issues/3013/files/test.cpp
@@ -0,0 +1,2 @@
+void some_function() {}
+
diff --git a/test/SideEffect/Issues/3013/sideffect_with_variantdir.py b/test/SideEffect/Issues/3013/sideffect_with_variantdir.py
new file mode 100644
--- /dev/null
+++ b/test/SideEffect/Issues/3013/sideffect_with_variantdir.py
@@ -0,0 +1,49 @@
+#!/usr/bin/env python
+#
+# __COPYRIGHT__
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+
+__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"
+
+"""
+Test materials for Github Issue 3013 submitted by Stefan Ross:
+https://github.com/SCons/scons/issues/3013
+"""
+
+import TestSCons
+
+test = TestSCons.TestSCons()
+
+test.dir_fixture('files')
+
+test.run(
+    arguments = '-j2'
+)
+
+test.pass_test()
+
+# Local Variables:
+# tab-width:4
+# indent-tabs-mode:nil
+# End:
+# vim: set expandtab tabstop=4 shiftwidth=4:
+