Repository: SCons/scons

Issue:
SCons does not remove dangling symlink target before build
Hi

According to https://scons.org/doc/3.1.2/HTML/scons-user/ch13.html, _when SCons determines that an target file needs to be rebuilt and removes the existing version of the target before executing_.

Unfortunately the above is not true, when the target file for some reason is a symbolic link to non-existing file. In such a case scons leaves the broken symlink as is, and then runs target build action, which then might try to create target file in the destination of the symlink (for example g++ works like this).

Is there any possibility to convince scons to remove the target's dangling symlink before starting  build?



Discussion:
Do you have a small test case to demonstrate this issue?

Yes, I'm attaching a simple test case with two broken symlinks, that are not removed by scons, what causes g++ to follow them. 
[test-case.tar.gz](https://github.com/SCons/scons/files/4027818/test-case.tar.gz)

In your test case the symlink isn't created by SCons.. 
Is that the actual situation you're working with? Or is SCons creating the symlink for you?
Hi,

What I am trying to do is using symlinks to files in cache instead of copying them. (I know hardlinks would be much better, but the cache is on separate storage, so they can't be used.) The aim is to have better performance and lower disk usage. 

Anyway, I've already created my own versions of  SCons.CacheDir.CacheRetrieve & CacheRetrieveSilent actions, in which I create those symlinks. I even remove dangling symlinks from there as well. 

I'm still testing this solution, but everything seems to work pretty well so far unless the --cache-disable option is used, after the link target had already been removed, for example by a cron job that clears old files from the cache. In such a case g++ happily creates new file via the symlink in the cache. That's why I am seeking for solution that would allow me to remove the broken symlinks  before g++ is run. 

I've recently learnt about AddPreAction(), but I am not sure if is it possible to add such action for any target. I do not know scons so well yet to get a list of all targets. But maybe it would be simpler if scons could somehow  remove such broken links on its own? I think it must use os.path.exists() somewhere; maybe it would be easy to use to replace it with  os.path.lexists()?

Thanks,
Robert

Resolving patch:
diff --git a/SCons/Node/FS.py b/SCons/Node/FS.py
--- a/SCons/Node/FS.py
+++ b/SCons/Node/FS.py
@@ -710,6 +710,20 @@ def stat(self):
         self._memo['stat'] = result
         return result
 
+    @SCons.Memoize.CountMethodCall
+    def lstat(self):
+        try:
+            return self._memo['lstat']
+        except KeyError:
+            pass
+        try:
+            result = self.fs.lstat(self.get_abspath())
+        except os.error:
+            result = None
+
+        self._memo['lstat'] = result
+        return result
+
     def exists(self):
         return SCons.Node._exists_map[self._func_exists](self)
 
@@ -717,14 +731,22 @@ def rexists(self):
         return SCons.Node._rexists_map[self._func_rexists](self)
 
     def getmtime(self):
-        st = self.stat()
+        if self.islink():
+            st = self.lstat()
+        else:
+            st = self.stat()
+
         if st:
             return st[stat.ST_MTIME]
         else:
             return None
 
     def getsize(self):
-        st = self.stat()
+        if self.islink():
+            st = self.lstat()
+        else:
+            st = self.stat()
+
         if st:
             return st[stat.ST_SIZE]
         else:
@@ -3103,7 +3125,10 @@ def prepare(self):
         SCons.Node.Node.prepare(self)
 
         if self.get_state() != SCons.Node.up_to_date:
-            if self.exists():
+            # Exists will report False for dangling symlinks so if it
+            # exists or is a link (which would mean it's a dangling
+            # link) then we should remove it as appropriate.
+            if self.exists() or self.islink():
                 if self.is_derived() and not self.precious:
                     self._rmv_existing()
             else:
diff --git a/SCons/Node/FSTests.py b/SCons/Node/FSTests.py
--- a/SCons/Node/FSTests.py
+++ b/SCons/Node/FSTests.py
@@ -32,7 +32,7 @@
 import shutil
 import stat
 
-from TestCmd import TestCmd
+from TestCmd import TestCmd, IS_WINDOWS
 import TestUnit
 
 import SCons.Errors
@@ -3541,6 +3541,65 @@ def __call__(self, target, source, env, executor=None):
         dir = fs.Dir("dir")
         dir.prepare()
 
+@unittest.skipIf(IS_WINDOWS, "No symlinks on windows")
+@unittest.skipUnless(hasattr(os, 'symlink'), "Platform doesn't support symlink")
+class CleanSymlinksTestCase(_tempdirTestCase):
+
+    def test_cleans_symlinks(self):
+        """Test the prepare() method will cleanup symlinks."""
+
+        test = self.test
+
+        with open(test.workpath("foo"), "w") as foo:
+            foo.write("baz")
+
+        os.symlink(test.workpath("foo"), test.workpath("bar"))
+        bar = self.fs.File(test.workpath("bar"))
+        bar.side_effect = True
+        bar.set_state(0)
+        assert bar.exists(), "Symlink %s should not exist after prepare"%str(bar)
+
+        bar.prepare()
+        try:
+            os.lstat(test.workpath("bar"))
+            assert False, "bar should not exist"
+        except FileNotFoundError:
+            pass
+
+        try:
+            os.stat(test.workpath("foo"))
+        except FileNotFoundError:
+            test.fail('Real file %s should not be removed'%test.workpath('foo'))
+
+    def test_cleans_dangling_symlinks(self):
+        """Test the prepare() method will cleanup dangling symlinks."""
+        test = self.test
+
+        with open(test.workpath("foo"), "w") as foo:
+            foo.write("baz")
+
+        os.symlink(test.workpath("foo"), test.workpath("bar"))
+        os.remove(test.workpath("foo"))
+        try:
+            os.stat(test.workpath("foo"))
+            assert False, "foo should not exist"
+        except FileNotFoundError:
+            pass
+
+        bar = self.fs.File(test.workpath("bar"))
+        bar.side_effect = True
+        bar.set_state(0)
+
+        # Dangling links should report not exists
+        assert not bar.exists()
+
+        bar.prepare()
+        try:
+            os.lstat(test.workpath("bar"))
+            assert False, "bar [%s] should not exist"%test.workpath("bar")
+        except FileNotFoundError:
+            pass
+
 
 class SConstruct_dirTestCase(unittest.TestCase):
     def runTest(self):
diff --git a/runtest.py b/runtest.py
--- a/runtest.py
+++ b/runtest.py
@@ -10,7 +10,7 @@
 #  - end-to-end tests  - these are *.py files in test/ directory that
 #                        require custom SCons framework from testing/
 #
-# This script adds src/ and testing/ directories to PYTHONPATH,
+# This script adds SCons/ and testing/ directories to PYTHONPATH,
 # performs test discovery and processes them according to options.
 #
 # With -p (--package) option, script tests specified package from
@@ -603,18 +603,18 @@ def find_e2e_tests(directory):
     testpaths = []
 
     # Each test path specifies a test file, or a directory to search for
-    # SCons tests. SCons code layout assumes that any file under the 'src'
+    # SCons tests. SCons code layout assumes that any file under the 'SCons'
     # subdirectory that ends with 'Tests.py' is a unit test, and any Python
     # script (*.py) under the 'test' subdirectory an end-to-end test.
     # We need to track these because they are invoked differently.
     #
-    # Note that there are some tests under 'src' that *begin* with
+    # Note that there are some tests under 'SCons' that *begin* with
     # 'test_', but they're packaging and installation tests, not
     # functional tests, so we don't execute them by default.  (They can
     # still be executed by hand, though).
 
     if options.all:
-        testpaths = ['src', 'test']
+        testpaths = ['SCons', 'test']
     elif args:
         testpaths = args
 
@@ -629,7 +629,7 @@ def find_e2e_tests(directory):
         # sys.stderr.write("to:%s\n"%tp)
         for path in glob.glob(tp):
             if os.path.isdir(path):
-                if path.startswith('src') or path.startswith('testing'):
+                if path.startswith('SCons') or path.startswith('testing'):
                     for p in find_unit_tests(path):
                         unittests.append(p)
                 elif path.startswith('test'):


Test patch:
diff --git a/test/runtest/src.py b/test/runtest/SCons.py
similarity index 76%
rename from test/runtest/src.py
rename to test/runtest/SCons.py
--- a/test/runtest/src.py
+++ b/test/runtest/SCons.py
@@ -25,7 +25,7 @@
 __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"
 
 """
-Verify that we find tests under the src/ tree only if they end
+Verify that we find tests under the SCons/ tree only if they end
 with *Tests.py.
 """
 
@@ -35,21 +35,21 @@
 
 test = TestRuntest.TestRuntest()
 
-test.subdir(['src'],
-            ['src', 'suite'])
+test.subdir(['SCons'],
+            ['SCons', 'suite'])
 
 pythonstring = TestRuntest.pythonstring
 pythonflags = TestRuntest.pythonflags
-src_passTests_py = os.path.join('src', 'passTests.py')
-src_suite_passTests_py = os.path.join('src', 'suite', 'passTests.py')
+src_passTests_py = os.path.join('SCons', 'passTests.py')
+src_suite_passTests_py = os.path.join('SCons', 'suite', 'passTests.py')
 
-test.write_passing_test(['src', 'pass.py'])
+test.write_passing_test(['SCons', 'pass.py'])
 
-test.write_passing_test(['src', 'passTests.py'])
+test.write_passing_test(['SCons', 'passTests.py'])
 
-test.write_passing_test(['src', 'suite', 'pass.py'])
+test.write_passing_test(['SCons', 'suite', 'pass.py'])
 
-test.write_passing_test(['src', 'suite', 'passTests.py'])
+test.write_passing_test(['SCons', 'suite', 'passTests.py'])
 
 expect_stdout = """\
 %(pythonstring)s%(pythonflags)s %(src_passTests_py)s
@@ -63,7 +63,7 @@
 PASSING TEST STDERR
 """ % locals()
 
-test.run(arguments='-k src', stdout=expect_stdout, stderr=expect_stderr)
+test.run(arguments='-k SCons', stdout=expect_stdout, stderr=expect_stderr)
 
 test.pass_test()