Repository: SCons/scons

Issue:
Methods added to environments with Scons.Util.AddMethod do not env.Clone correctly
This issue was originally created at: 2016-05-03 07:51:15.
This issue was reported by: `pw201`.
pw201 said at 2016-05-03 07:51:15
>Methods added to an environment instance with the SCons.Util.AddMethod 
>function do not clone correctly with env.Clone(): they're bound to the old 
>object, not the cloned one. If you use the AddMethod method on the instance 
>itself, this works, because the method adds them to the instance's 
>added_method's list. 
> 
>Although the example in the docs uses the AddMethod function on a class and 
>the AddMethod method on an instance of the class, as the function is 
>documented to work on an "object", it's not clear that this shouldn't work 
>on an instance of Environment rather than only working on the Environment 
>class.
> 
>That is, I assert that given
> 
>def funct(self):
>   return self
> 
>then
> 
>env = Environment()
>AddMethod(env, funct)
>env1 = env.Clone()
>print env1, env1.funct()
> 
>and 
> 
>env = Environment()
>env.AddMethod(funct)
>env1 = env.Clone()
>print env1, env1.funct()
> 
>should both print the same object twice (or the first example should 
>generate an exception when AddMethod is used on an instance rather than a 
>class). Currently only the second example does so.

bdbaddog said at 2016-05-03 10:35:07
>Thanks for the bug report.
>Please bring your issue to the scons users mailing list before filing a bug.
>This is the standard scons policy.  The idea is to avoid filing duplicate bugs,
>get more eyes on the issue which may provide a faster fix and/or work around. 
>Also this helps identify issues which are user error and not issues with SCons.
> 
>http://scons.org/bugs.html
> 
>While it looks like your issue is a new one, please still bring this to the
>users mailing list.

pw201 said at 2016-05-04 09:30:05
>Sorry, but the page at http://scons.tigris.org/bug-submission.html doesn't 
>mention the mailing list. I won't be subscribing to lots of mails just to 
>report a bug. I have provided a clear statement of the problem and a 
>reproduction case. Feel free to fix it or not.

bdbaddog said at 2016-05-04 16:57:15
>Fair enough. Though the volume of email on the users mailing list is not that
>high.  (as you can see here:
>http://news.gmane.org/gmane.comp.programming.tools.scons.user)
> 
>I'll update the page to indicate the preferred process for reporting a bug.
>I'll pass along a link to our users mailing list as well to get more eyes on it.
> 
>Thanks for reporting the issue.




Discussion:
This  is interesting... and the behavior of `env.AddMethod()` seems to be pretty intentional. When called, it's not the method itself that is added, but a `MethodWrapper` instance, which stores the method. Then when there's a Clone, the act of cloning creates a new wrapper which will hold the same method. 

Expanding the bits of the original example, the `env.AddMethod` case:
```
env = Environment()
env.AddMethod(funct)
print("Added to:", env, "=", env.funct)
print("which wraps:", env.funct.method)
env1 = env.Clone()
print("Cloned to", env1, "=", env1.funct)
print("which wraps:", env.funct.method)
```
Which gives us:
```
Funct at <function funct at 0x7f16f5361268>
...
Added to: <SCons.Script.SConscript.SConsEnvironment object at 0x7f16f5236dd8> = <SCons.Environment.MethodWrapper object at 0x7f16f523c048>
which wraps: <function funct at 0x7f16f5361268>
Cloned to <SCons.Script.SConscript.SConsEnvironment object at 0x7f16f523c2b0> = <SCons.Environment.MethodWrapper object at 0x7f16f51db048>
which wraps: <function funct at 0x7f16f5361268>
```
So the question is, is the different behavior of `Clone()` intentional?  Here we don't get `MethodWrapper` objects, we get methods bound into the environment they are added to, as if they were methods of that environment in other words.
```
print("Funct at", funct)
env = Environment()
AddMethod(env, funct)
print("Added to:", env, env.funct)
env1 = env.Clone()
print("Cloned to:", env1, env1.funct)
```
Giving:
```
Funct at <function funct at 0x7f16f5361268>
Added to: <SCons.Script.SConscript.SConsEnvironment object at 0x7f16f538a1d0> <bound method funct of <SCons.Script.SConscript.SConsEnvironment object at 0x7f16f538a1d0>>
Cloned to: <SCons.Script.SConscript.SConsEnvironment object at 0x7f16f538a470> <bound method funct of <SCons.Script.SConscript.SConsEnvironment object at 0x7f16f538a1d0>>
```
In any case, `Clone` and `env.Clone` are different code paths, one is in `SCons.Util` and the other in `SCons.Environment`
Inconsistent, but is this actually a problem, in the end?
Swung back and took another look.  `AddMethod` (`SCons.Util`) and `env.AddMethod` (`SCons.Environment`) are different functions.  The latter appends any added methods to an attribute named, helpfully, `added_methods` in the instance it's acting on.  This attribute is then used when cloning to rebind added methods.  If you use `AddMethod` this never happens, so there's no rebinding. Perhaps fix this in documentation ("do not try to AddMethod to an Environment instance, use env.AddMethod"  kind of thing)? It looked like a fairly trivial fix in `SCons.Util.AddMethod` but it turns out you don't bind the right thing in that case, and trying to get it from `SCons.Environment` leads to a circular import warning that I don't feel like unraveling.
How about a warning when if you do that? ( `if isinstanct(BaseEnvironment, object0)` ?)

A warning might make sense... what would the wording be?   When I tried a quick hack I looked for whether the object had a `added_methods` attribute, since that's set in the object initializer, which detected the case nicely enough, I just couldn't "fix things up" easily at that point.
Hmm.. Is it always bad to use AddMethod(env,...) ? Or only if you expect it to propagate when you Clone()?
If you move the import into the function involved (and thus narrow it's scope) I think that can work around circular import issues..
If you cloned, the wrong thing gets passed as "self" (you end up with the proxy for the cloned parent).  That may or may not be an actual problem (I presume at least something or this issue wouldn't have been filed). 
Don't think we can "fix" the imports, Util just can't import the other bits of scons, which basically all use something from Util. Could maybe move the definition of MethodWrapper to Util...

Ah..yes that'd sometimes be BAD.. but often o.k. which is kinda worse than always bad or good.. ;)
It does become a simple fix with MethodWrapper moved over. (I have a testcase for the unit test) Whether it's a good idea or not to even try... 

Resolving patch:
diff --git a/SCons/Environment.py b/SCons/Environment.py
--- a/SCons/Environment.py
+++ b/SCons/Environment.py
@@ -54,6 +54,7 @@
 import SCons.Subst
 import SCons.Tool
 import SCons.Util
+from SCons.Util import MethodWrapper
 import SCons.Warnings
 
 class _Null:
@@ -180,48 +181,17 @@ def _delete_duplicates(l, keep_last):
 # Shannon at the following page (there called the "transplant" class):
 #
 # ASPN : Python Cookbook : Dynamically added methods to a class
-# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81732
+# https://code.activestate.com/recipes/81732/
 #
 # We had independently been using the idiom as BuilderWrapper, but
 # factoring out the common parts into this base class, and making
 # BuilderWrapper a subclass that overrides __call__() to enforce specific
 # Builder calling conventions, simplified some of our higher-layer code.
+#
+# Note: MethodWrapper moved to SCons.Util as it was needed there
+# and otherwise we had a circular import problem.
 
-class MethodWrapper:
-    """
-    A generic Wrapper class that associates a method (which can
-    actually be any callable) with an object.  As part of creating this
-    MethodWrapper object an attribute with the specified (by default,
-    the name of the supplied method) is added to the underlying object.
-    When that new "method" is called, our __call__() method adds the
-    object as the first argument, simulating the Python behavior of
-    supplying "self" on method calls.
-
-    We hang on to the name by which the method was added to the underlying
-    base class so that we can provide a method to "clone" ourselves onto
-    a new underlying object being copied (without which we wouldn't need
-    to save that info).
-    """
-    def __init__(self, object, method, name=None):
-        if name is None:
-            name = method.__name__
-        self.object = object
-        self.method = method
-        self.name = name
-        setattr(self.object, name, self)
-
-    def __call__(self, *args, **kwargs):
-        nargs = (self.object,) + args
-        return self.method(*nargs, **kwargs)
-
-    def clone(self, new_object):
-        """
-        Returns an object that re-binds the underlying "method" to
-        the specified new object.
-        """
-        return self.__class__(new_object, self.method, self.name)
-
-class BuilderWrapper(MethodWrapper):
+class BuilderWrapper(SCons.Util.MethodWrapper):
     """
     A MethodWrapper subclass that that associates an environment with
     a Builder.
@@ -248,7 +218,7 @@ def __call__(self, target=None, source=_null, *args, **kw):
             target = [target]
         if source is not None and not SCons.Util.is_List(source):
             source = [source]
-        return MethodWrapper.__call__(self, target, source, *args, **kw)
+        return super().__call__(target, source, *args, **kw)
 
     def __repr__(self):
         return '<BuilderWrapper %s>' % repr(self.name)
@@ -2377,7 +2347,7 @@ def __getattr__(self, name):
         # Environment they are being constructed with and so will not
         # have access to overrided values. So we rebuild them with the
         # OverrideEnvironment so they have access to overrided values.
-        if isinstance(attr, (MethodWrapper, BuilderWrapper)):
+        if isinstance(attr, MethodWrapper):
             return attr.clone(self)
         else:
             return attr
diff --git a/SCons/EnvironmentTests.py b/SCons/EnvironmentTests.py
--- a/SCons/EnvironmentTests.py
+++ b/SCons/EnvironmentTests.py
@@ -33,7 +33,13 @@
 import TestCmd
 import TestUnit
 
-from SCons.Environment import *
+from SCons.Environment import (
+    Environment,
+    NoSubstitutionProxy,
+    OverrideEnvironment,
+    SubstitutionEnvironment,
+    is_valid_construction_var,
+)
 import SCons.Warnings
 
 def diff_env(env1, env2):
@@ -721,7 +727,7 @@ def func2(self, arg=''):
         r = env4.func2()
         assert r == 'func2-4', r
 
-        # Test that clones don't re-bind an attribute that the user
+        # Test that clones don't re-bind an attribute that the user set.
         env1 = Environment(FOO = '1')
         env1.AddMethod(func2)
         def replace_func2():
@@ -731,6 +737,18 @@ def replace_func2():
         r = env2.func2()
         assert r == 'replace_func2', r
 
+        # Test clone rebinding if using global AddMethod.
+        env1 = Environment(FOO='1')
+        SCons.Util.AddMethod(env1, func2)
+        r = env1.func2()
+        assert r == 'func2-1', r
+        r = env1.func2('-xxx')
+        assert r == 'func2-1-xxx', r
+        env2 = env1.Clone(FOO='2')
+        r = env2.func2()
+        assert r == 'func2-2', r
+
+
     def test_Override(self):
         """Test overriding construction variables"""
         env = SubstitutionEnvironment(ONE=1, TWO=2, THREE=3, FOUR=4)
diff --git a/SCons/Util.py b/SCons/Util.py
--- a/SCons/Util.py
+++ b/SCons/Util.py
@@ -27,23 +27,19 @@
 import sys
 import copy
 import re
-import types
 import pprint
 import hashlib
 from collections import UserDict, UserList, UserString, OrderedDict
 from collections.abc import MappingView
+from types import MethodType, FunctionType
 
 PYPY = hasattr(sys, 'pypy_translation_info')
 
-# Below not used?
-# InstanceType    = types.InstanceType
-
-MethodType      = types.MethodType
-FunctionType    = types.FunctionType
-
-def dictify(keys, values, result={}):
-    for k, v in zip(keys, values):
-        result[k] = v
+# unused?
+def dictify(keys, values, result=None):
+    if result is None:
+        result = {}
+    result.update(dict(zip(keys, values)))
     return result
 
 _altsep = os.altsep
@@ -633,6 +629,41 @@ def __get__(self, obj, cls):
         else:
             return self
 
+
+class MethodWrapper:
+    """A generic Wrapper class that associates a method with an object.
+
+    As part of creating this MethodWrapper object an attribute with the
+    specified name (by default, the name of the supplied method) is added
+    to the underlying object.  When that new "method" is called, our
+    __call__() method adds the object as the first argument, simulating
+    the Python behavior of supplying "self" on method calls.
+
+    We hang on to the name by which the method was added to the underlying
+    base class so that we can provide a method to "clone" ourselves onto
+    a new underlying object being copied (without which we wouldn't need
+    to save that info).
+    """
+    def __init__(self, object, method, name=None):
+        if name is None:
+            name = method.__name__
+        self.object = object
+        self.method = method
+        self.name = name
+        setattr(self.object, name, self)
+
+    def __call__(self, *args, **kwargs):
+        nargs = (self.object,) + args
+        return self.method(*nargs, **kwargs)
+
+    def clone(self, new_object):
+        """
+        Returns an object that re-binds the underlying "method" to
+        the specified new object.
+        """
+        return self.__class__(new_object, self.method, self.name)
+
+
 # attempt to load the windows registry module:
 can_read_reg = 0
 try:
@@ -1096,7 +1127,7 @@ def adjustixes(fname, pre, suf, ensure_suffix=False):
 
 
 # From Tim Peters,
-# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560
+# https://code.activestate.com/recipes/52560
 # ASPN: Python Cookbook: Remove duplicates from a sequence
 # (Also in the printed Python Cookbook.)
 
@@ -1170,9 +1201,8 @@ def unique(s):
     return u
 
 
-
 # From Alex Martelli,
-# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560
+# https://code.activestate.com/recipes/52560
 # ASPN: Python Cookbook: Remove duplicates from a sequence
 # First comment, dated 2001/10/13.
 # (Also in the printed Python Cookbook.)
@@ -1375,42 +1405,41 @@ def make_path_relative(path):
     return path
 
 
-
-# The original idea for AddMethod() and RenameFunction() come from the
+# The original idea for AddMethod() came from the
 # following post to the ActiveState Python Cookbook:
 #
-#   ASPN: Python Cookbook : Install bound methods in an instance
-#   http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/223613
-#
-# That code was a little fragile, though, so the following changes
-# have been wrung on it:
+# ASPN: Python Cookbook : Install bound methods in an instance
+# https://code.activestate.com/recipes/223613
 #
+# Changed as follows:
 # * Switched the installmethod() "object" and "function" arguments,
 #   so the order reflects that the left-hand side is the thing being
 #   "assigned to" and the right-hand side is the value being assigned.
-#
-# * Changed explicit type-checking to the "try: klass = object.__class__"
-#   block in installmethod() below so that it still works with the
-#   old-style classes that SCons uses.
-#
-# * Replaced the by-hand creation of methods and functions with use of
-#   the "new" module, as alluded to in Alex Martelli's response to the
-#   following Cookbook post:
-#
-#   ASPN: Python Cookbook : Dynamically added methods to a class
-#   http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81732
+# * The instance/class detection is changed a bit, as it's all
+#   new-style classes now with Py3.
+# * The by-hand construction of the function object from renamefunction()
+#   is not needed, the remaining bit is now used inline in AddMethod.
 
 def AddMethod(obj, function, name=None):
-    """
-    Adds either a bound method to an instance or the function itself (or an unbound method in Python 2) to a class.
-    If name is ommited the name of the specified function
-    is used by default.
+    """Adds a method to an object.
+
+    Adds `function` to `obj` if `obj` is a class object.
+    Adds `function` as a bound method if `obj` is an instance object.
+    If `obj` looks like an environment instance, use `MethodWrapper`
+    to add it.  If `name` is supplied it is used as the name of `function`.
+
+    Although this works for any class object, the intent as a public
+    API is to be used on Environment, to be able to add a method to all
+    construction environments; it is preferred to use env.AddMethod
+    to add to an individual environment.
 
     Example::
 
+        class A:
+            ...
         a = A()
         def f(self, x, y):
-        self.z = x + y
+            self.z = x + y
         AddMethod(f, A, "add")
         a.add(2, 4)
         print(a.z)
@@ -1420,32 +1449,24 @@ def f(self, x, y):
     if name is None:
         name = function.__name__
     else:
-        function = RenameFunction(function, name)
+        # "rename"
+        function = FunctionType(
+            function.__code__, function.__globals__, name, function.__defaults__
+        )
 
-    # Note the Python version checks - WLB
-    # Python 3.3 dropped the 3rd parameter from types.MethodType
     if hasattr(obj, '__class__') and obj.__class__ is not type:
-        # "obj" is an instance, so it gets a bound method.
-        if sys.version_info[:2] > (3, 2):
-            method = MethodType(function, obj)
+        # obj is an instance, so it gets a bound method.
+        if hasattr(obj, "added_methods"):
+            method = MethodWrapper(obj, function, name)
+            obj.added_methods.append(method)
         else:
-            method = MethodType(function, obj, obj.__class__)
+            method = MethodType(function, obj)
     else:
-        # Handle classes
+        # obj is a class
         method = function
 
     setattr(obj, name, method)
 
-def RenameFunction(function, name):
-    """
-    Returns a function identical to the specified function, but with
-    the specified name.
-    """
-    return FunctionType(function.__code__,
-                        function.__globals__,
-                        name,
-                        function.__defaults__)
-
 
 if hasattr(hashlib, 'md5'):
     md5 = True
@@ -1523,8 +1544,7 @@ def silent_intern(x):
 
 # From Dinu C. Gherman,
 # Python Cookbook, second edition, recipe 6.17, p. 277.
-# Also:
-# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/68205
+# Also: https://code.activestate.com/recipes/68205
 # ASPN: Python Cookbook: Null Object Design Pattern
 
 class Null:


Test patch: