Repository: SCons/scons

Issue:
MD5-timestamp decider does no longer follow symlinks
I reported the issue to Debian yesterday, see https://bugs.debian.org/981584,  and was asked by @bdbaddog to file it here

**Describe the bug**

I'm attaching a simple [scons-test-project.tar.gz](https://github.com/SCons/scons/files/5910894/scons-test-project.tar.gz) that:
- contains hello2.c file that is symbolic link to hello.c file;
- and sets 'MD5-timestamp' as the decider function.

Previously (what I'm proving at the bottom of this mail) scons used
to build hello2 when hello.c was changed. In the current version of 
scons only hello gets rebuilt.

Initial compilation with scons 4:
```
  robert@vox:/tmp/proj$ scons --debug=explain
  scons: Reading SConscript files ...
  scons: done reading SConscript files.
  scons: Building targets ...
  scons: building `hello.o' because it doesn't exist
  gcc -o hello.o -c hello.c
  scons: building `hello' because it doesn't exist
  gcc -o hello hello.o
  scons: building `hello2.o' because it doesn't exist
  gcc -o hello2.o -c hello2.c
  scons: building `hello2' because it doesn't exist
  gcc -o hello2 hello2.o
  scons: done building targets.
```
The following command changes both hello.c and hello2.c...  
```
  robert@vox:/tmp/proj$ sed -i -e 's/word/WoRd/i' hello.c
```
... but hello2.c is not rebuilt:
```
  robert@vox:/tmp/proj$ scons --debug=explain
  scons: Reading SConscript files ...
  scons: done reading SConscript files.
  scons: Building targets ...
  scons: rebuilding `hello.o' because:
             `hello.c' changed
             `/usr/bin/gcc' changed
  gcc -o hello.o -c hello.c
  scons: rebuilding `hello' because:
             `hello.o' changed
             `/usr/bin/gcc' changed
  gcc -o hello hello.o
  scons: done building targets.

  robert@vox:/tmp/proj$ scons -v
  SCons by Steven Knight et al.:
  	SCons: v4.0.1.c289977f8b34786ab6c334311e232886da7e8df1, 2020-07-17 01:50:03, by bdbaddog on ProDog2020
  	SCons path: ['/usr/lib/python3/dist-packages/SCons']
  Copyright (c) 2001 - 2020 The SCons Foundation
```


After downgrading scons to 3.1.2+dfsg-0.1, it works as I would expect:

Initial compilation with scons 3:
```
  robert@vox:/tmp/proj$ scons --debug=explain
  scons: Reading SConscript files ...
  scons: done reading SConscript files.
  scons: Building targets ...
  scons: building `hello.o' because it doesn't exist
  gcc -o hello.o -c hello.c
  scons: building `hello' because it doesn't exist
  gcc -o hello hello.o
  scons: building `hello2.o' because it doesn't exist
  gcc -o hello2.o -c hello2.c
  scons: building `hello2' because it doesn't exist
  gcc -o hello2 hello2.o
  scons: done building targets.
```  
Still both files are changed by the following command:
```
  robert@vox:/tmp/proj$ sed -i -e 's/word/WORd/i' hello.c
```
 ... and both are rebuilt:
```
  robert@vox:/tmp/proj$ scons --debug=explain
  scons: Reading SConscript files ...
  scons: done reading SConscript files.
  scons: Building targets ...
  scons: rebuilding `hello.o' because:
             `hello.c' changed
             `/usr/bin/gcc' changed
  gcc -o hello.o -c hello.c
  scons: rebuilding `hello' because:
             `hello.o' changed
             `/usr/bin/gcc' changed
  gcc -o hello hello.o
  scons: rebuilding `hello2.o' because:
             `hello2.c' changed
             `/usr/bin/gcc' changed
  gcc -o hello2.o -c hello2.c
  scons: rebuilding `hello2' because:
             `hello2.o' changed
             `/usr/bin/gcc' changed
  gcc -o hello2 hello2.o
  scons: done building targets.

  robert@vox:/tmp/proj$  scons -v
  SCons by Steven Knight et al.:
  	script: v3.1.2.bee7caf9defd6e108fc2998a2520ddb36a967691, 2019-12-17 02:07:09, by bdeegan on octodog
  	engine: v3.1.2.bee7caf9defd6e108fc2998a2520ddb36a967691, 2019-12-17 02:07:09, by bdeegan on octodog
  	engine path: ['/usr/lib/scons/SCons']
  Copyright (c) 2001 - 2019 The SCons Foundation
```

**Required information**
* Link to SCons Users thread discussing your issue.
* Version of SCons: 4.01,  but 4.10 seems to have the same bug as well
* Version of Python: 3.9
* Which python distribution if applicable (python.org, cygwin, anaconda, macports, brew,etc)
* How you installed SCons
* What Platform are you on? (Linux/Windows and which version) Debian GNU/Linux
* How to reproduce your issue?  Please include a small self contained reproducer. Likely a SConstruct should do for most issues.
* How you invoke scons (The command line you're using "scons --flags some_arguments")



Discussion:
Thanks for posting the issue.
We just hit this bug in 4.3.0 after coming from scons-3.1.2, and it caused a lot of havoc in our company.  We have disabled MD5-timestamp for now and will suffer the compile-time hit.
probably this commit: https://github.com/SCons/scons/commit/6cde83c2b3fe94eb7c719abb4426a10592ef48d9
It's frustrating that the explain code also produces deceptive information (the bit about gcc changing is just wrong), but that is a different problem - unfortunately the chosen decider is not the section of code that produces the explantion, that's its own function.

Meanwhile, the commit pointed to looks dodgy to me - the `getmtime` function at line 734 will then return the modification time of the link itself, and thus not see that what it links to has an updated timestamp.  Or am I completely missing the point?
> Meanwhile, the commit pointed to looks dodgy to me - the getmtime function at line 734 will then return the modification time of the link itself, and thus not see that what it links to has an updated timestamp. Or am I completely missing the point?

Yeah, probably should not have the changes to getmtime and getsize.
I'd say this wants some more eyeballs.  #3638 was made to close #3516, problems with dangling symlinks as targets, but it seems to have left behind this problem with sources that are symlinks. As a quick test, If I remove the expanded code in the `getmtime` function here:

https://github.com/SCons/scons/blob/57369e87ba560f941e645e9d2ed194bd4a26dad5/SCons/Node/FS.py#L735

the detection of the symlinked target starts working again (for the test in this issue, `hello2.o` is rebuilt). There *was* a unit test added for #3638, which can be seen here:

https://github.com/SCons/scons/blob/57369e87ba560f941e645e9d2ed194bd4a26dad5/SCons/Node/FSTests.py#L3604

and that still passes after the change.  I haven't had the energy to think through the whole logic here, and it would be good to work up a unit test for this case as well, so what do folks think?
It's clear that the MD5 symlink behavior got broken and needs to be fixed (and a unit test added).  However, is pre-clearing broken symlinks a desired behavior in the first place?  My reference standard is make.  Here is an example that compares behavior between make, scons-3.1.2, and scons-4.3.0.  Note that "scons-3.1.2 --clean" does the "wrong" thing, in my opinion.  scons should only clean the actual files that it generated (restoring the original system state as much as possible). 

[scons-symlink.tar.gz](https://github.com/SCons/scons/files/8309782/scons-symlink.tar.gz)
 
> It's clear that the MD5 symlink behavior got broken and needs to be fixed (and a unit test added). However, is pre-clearing broken symlinks a desired behavior in the first place? My reference standard is make. Here is an example that compares behavior between make, scons-3.1.2, and scons-4.3.0. Note that "scons-3.1.2 --clean" does the "wrong" thing, in my opinion. scons should only clean the actual files that it generated (restoring the original system state as much as possible).
> 
> [scons-symlink.tar.gz](https://github.com/SCons/scons/files/8309782/scons-symlink.tar.gz)

Yes, (at least minimally) if SCons creates those symlinks.
SCons is not limited to the functionality of make in our scope.  (by that measure md5 deciders would also not be viable.. ;)

> Yes, (at least minimally) if SCons creates those symlinks.
> SCons is not limited to the functionality of make in our scope. (by that measure md5 deciders would also not be viable.. ;)

I absolutely agree.  After further experimentation and thought, the current behavior of replacing the symlink is sane, and I withdraw my pre-clearing question.  Trying to figure out a spec of when and when not to replace symlinks drove me insane :-).
Yup. it's tricky. :)
Welcome to some of the guts of SCons.. ;)
Suggestions for a test?  I'm not seeing where to put something in the unit tests, though there is a `test/symlink` directory in the e2e tests.
It would be along the lines of @rodrigc's testcase.  Make sure a file is rebuilt.

Resolving patch:
diff --git a/SCons/Node/FS.py b/SCons/Node/FS.py
--- a/SCons/Node/FS.py
+++ b/SCons/Node/FS.py
@@ -255,7 +255,7 @@ def _softlink_func(fs, src, dst):
 def _copy_func(fs, src, dest):
     shutil.copy2(src, dest)
     st = fs.stat(src)
-    fs.chmod(dest, stat.S_IMODE(st[stat.ST_MODE]) | stat.S_IWRITE)
+    fs.chmod(dest, stat.S_IMODE(st.st_mode) | stat.S_IWRITE)
 
 
 Valid_Duplicates = ['hard-soft-copy', 'soft-hard-copy',
@@ -733,10 +733,7 @@ def rexists(self):
         return SCons.Node._rexists_map[self._func_rexists](self)
 
     def getmtime(self):
-        if self.islink():
-            st = self.lstat()
-        else:
-            st = self.stat()
+        st = self.stat()
 
         if st:
             return st[stat.ST_MTIME]
@@ -744,28 +741,25 @@ def getmtime(self):
             return None
 
     def getsize(self):
-        if self.islink():
-            st = self.lstat()
-        else:
-            st = self.stat()
+        st = self.stat()
 
         if st:
-            return st[stat.ST_SIZE]
+            return st.st_size
         else:
             return None
 
     def isdir(self):
         st = self.stat()
-        return st is not None and stat.S_ISDIR(st[stat.ST_MODE])
+        return st is not None and stat.S_ISDIR(st.st_mode)
 
     def isfile(self):
         st = self.stat()
-        return st is not None and stat.S_ISREG(st[stat.ST_MODE])
+        return st is not None and stat.S_ISREG(st.st_mode)
 
     if hasattr(os, 'symlink'):
         def islink(self):
             st = self.lstat()
-            return st is not None and stat.S_ISLNK(st[stat.ST_MODE])
+            return st is not None and stat.S_ISLNK(st.st_mode)
     else:
         def islink(self):
             return False                    # no symlinks
@@ -1939,7 +1933,7 @@ def srcnode(self):
             return self.srcdir
         return Base.srcnode(self)
 
-    def get_timestamp(self):
+    def get_timestamp(self) -> int:
         """Return the latest timestamp from among our children"""
         stamp = 0
         for kid in self.children():
@@ -1947,11 +1941,11 @@ def get_timestamp(self):
                 stamp = kid.get_timestamp()
         return stamp
 
-    def get_abspath(self):
+    def get_abspath(self) -> str:
         """Get the absolute path of the file."""
         return self._abspath
 
-    def get_labspath(self):
+    def get_labspath(self) -> str:
         """Get the absolute path of the file."""
         return self._labspath
 
diff --git a/SCons/Node/FSTests.py b/SCons/Node/FSTests.py
--- a/SCons/Node/FSTests.py
+++ b/SCons/Node/FSTests.py
@@ -408,8 +408,8 @@ def Link_IOError(target, source, env):
                            None)
         os.chmod(test.workpath('src/foo'), stat.S_IRUSR | stat.S_IWRITE)
         st = os.stat(test.workpath('build/foo'))
-        assert (stat.S_IMODE(st[stat.ST_MODE]) & stat.S_IWRITE), \
-            stat.S_IMODE(st[stat.ST_MODE])
+        assert (stat.S_IMODE(st.st_mode) & stat.S_IWRITE), \
+            stat.S_IMODE(st.st_mode)
 
         # This used to generate a UserError when we forbid the source
         # directory from being outside the top-level SConstruct dir.
@@ -771,7 +771,7 @@ def test_update(self):
 
         mtime = st[stat.ST_MTIME]
         assert ni.timestamp == mtime, (ni.timestamp, mtime)
-        size = st[stat.ST_SIZE]
+        size = st.st_size
         assert ni.size == size, (ni.size, size)
 
         import time
@@ -783,7 +783,7 @@ def test_update(self):
 
         mtime = st[stat.ST_MTIME]
         assert ni.timestamp != mtime, (ni.timestamp, mtime)
-        size = st[stat.ST_SIZE]
+        size = st.st_size
         assert ni.size != size, (ni.size, size)
 
 


Test patch:
diff --git a/test/Decider/content-timestamp-symlink.py b/test/Decider/content-timestamp-symlink.py
new file mode 100644
--- /dev/null
+++ b/test/Decider/content-timestamp-symlink.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python
+#
+# Copyright The SCons Foundation
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+"""
+Test the content-timestamp decider (formerly known as md5-timestamp)
+correctly detects modification of a source file which is a symlink.
+"""
+
+import TestSCons
+
+_python_ = TestSCons._python_
+
+test = TestSCons.TestSCons()
+
+if not test.platform_has_symlink():
+    test.skip_test('Symbolic links not reliably available on this platform, skipping test.\n')
+
+# a dummy "compiler" for the builder
+test.write('build.py', r"""
+import sys
+
+with open(sys.argv[1], 'wb') as ofp:
+    for infile in sys.argv[2:]:
+        with open (infile, 'rb') as ifp:
+            ofp.write(ifp.read())
+sys.exit(0)
+""")
+
+test.write('SConstruct', """
+DefaultEnvironment(tools=[])
+Build = Builder(action=r'%(_python_)s build.py $TARGET $SOURCES')
+env = Environment(tools=[], BUILDERS={'Build': Build})
+env.Decider('content-timestamp')
+env.Build(target='match1.out', source='match1.in')
+env.Build(target='match2.out', source='match2.in')
+""" % locals())
+
+test.write('match1.in', 'match1.in\n')
+test.symlink('match1.in', 'match2.in')
+
+test.run(arguments='.')
+test.must_match('match1.out', 'match1.in\n')
+test.must_match('match2.out', 'match1.in\n')
+
+# first make sure some time has elapsed, so a low-granularity timestamp
+# doesn't fail to trigger
+test.sleep()
+# Now update the source file contents, both targets should rebuild
+test.write('match1.in', 'match2.in\n')
+
+test.run(arguments='.')
+test.must_match('match1.out', 'match2.in\n', message="match1.out not rebuilt\n")
+test.must_match('match2.out', 'match2.in\n', message="match2.out not rebuilt\n")
+
+test.pass_test()
+
+# Local Variables:
+# tab-width:4
+# indent-tabs-mode:nil
+# End:
+# vim: set expandtab tabstop=4 shiftwidth=4:
diff --git a/testing/framework/TestSCons.py b/testing/framework/TestSCons.py
--- a/testing/framework/TestSCons.py
+++ b/testing/framework/TestSCons.py
@@ -1665,7 +1665,12 @@ def get_alt_cpp_suffix(self):
             alt_cpp_suffix = '.C'
         return alt_cpp_suffix
 
-    def platform_has_symlink(self):
+    def platform_has_symlink(self) -> bool:
+        """Retun an indication of whether symlink tests should be run.
+
+        Despite the name, we really mean "are they reliably usable"
+        rather than "do they exist" - basically the Windows case.
+        """
         if not hasattr(os, 'symlink') or sys.platform == 'win32':
             return False
         else: