Repository: Pyomo/pyomo

Issue:
AMPLFUNC Environment Variable
It seems that when you call an ASL solver multiple times from a Python script, and you use ExternalFucntion, the AMPLFUNC environment variable accumulates a bunch of repeated copies of the .so file.  I haven't determined if this causes a real problem yet, but it causes the ASL to spew a bunch of warnings about already defined functions.  I printed the environment variables during an initialization consisting of several solves, so you can get an idea of what's happening

'PYOMO_AMPLFUNC': '../../../properties/physical/MEA_Simple/phys_prop.so'

'AMPLFUNC': '/imports/users/jeslick/git/eslickj/models/idaes_models/properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so'


Discussion:
Can you add a ticket for this?  This looks like a bug.

On Mon, Nov 21, 2016 at 1:38 PM, John Eslick notifications@github.com
wrote:

> It seems that when you call an ASL solver multiple times from a Python
> script, and you use ExternalFucntion, the AMPLFUNC environment variable
> accumulates a bunch of repeated copies of the .so file. I haven't
> determined if this causes a real problem yet, but it causes the ASL to spew
> a bunch of warnings about already defined functions. I printed the
> environment variables during an initialization consisting of several
> solves, so you can get an idea of what's happening
> 
> 'PYOMO_AMPLFUNC': '../../../properties/physical/MEA_Simple/phys_prop.so'
> 
> 'AMPLFUNC': '/imports/users/jeslick/git/eslickj/models/idaes_models/
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so'
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Pyomo/pyomo/issues/73, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AAsb-KXaYwWeP_eYDg7r0PRdpIidtggNks5rAeUXgaJpZM4K4jAD
> .

Sorry, I'm pretty new to how github works.  I made an issue.  How do I go about adding a ticket?
Ah.  I missed that this email was generated from an issue.  Thanks!

On Mon, Nov 21, 2016 at 1:47 PM, John Eslick notifications@github.com
wrote:

> Sorry, I'm pretty new to how github works. I made an issue. How do I go
> about adding a ticket?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/Pyomo/pyomo/issues/73#issuecomment-262029779, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AAsb-LWzBGWcVWVJij7vWZ0yFp4gBWIRks5rAedFgaJpZM4K4jAD
> .

Thanks for looking into it.  I've been having some trouble with the solver failing mysteriously, and after a lot of searching this seems to be the culprit.  I did some more testing, and after a certain number of times importing the .so file the ipopt solver segfaults before even getting started.  I have no idea why that would happen. It may be something weird in my user function code.  Anyway this bug does seem to potentially cause more problems than just annoying warning messages. 
To follow up on this issue, a couple other people working on IDAES have had problems with ipopt crashing when a library gets included in AMPLFUNC too many times, so this does seem to be an important issue.  Previously I wasn't totally sure whether or not this was a serious issue.
@eslickj: can you take a look at Pyomo/IDAES-premerge (240d3f9) and see if that resolves this issue?
@eslickj Has this been resolved?
I don't think so.  Just pulled the latest from Master. @jsiirola sorry I probably missed when you asked me to test this back in June.

Here is a bit of code I used to test.  Works if I set AMPLFUNC myself, but not if I don't. Not sure if you have a handy test for this, but I can get you the external function code if it helps.
```Python
from pyomo.environ import *
from pyomo.opt import SolverFactory
import os 

if __name__ == "__main__":
    model = ConcreteModel()
    flib = "./phys_prop.so"
    model.z_func = ExternalFunction(library=flib, function="z_func")
    model.x = Var(initialize=3)
    model.o = Objective(expr=model.z_func(model.x))
    opt = SolverFactory('ipopt')
    os.environ["AMPLFUNC"] = flib
    res = opt.solve(model)
    display(model)
```

Resolving patch:
diff --git a/pyomo/contrib/trustregion/getGJH.py b/pyomo/contrib/trustregion/getGJH.py
--- a/pyomo/contrib/trustregion/getGJH.py
+++ b/pyomo/contrib/trustregion/getGJH.py
@@ -56,9 +56,9 @@ def get_gjh(fname=None, insecure=False):
     os.chmod(fname, mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
 
 if __name__ == '__main__':
-    if len(sys.argv) > 1 and sys.argv[1] == '--insecure':
+    if len(sys.argv) > 1 and '--insecure' in sys.argv:
         insecure = True
-        sys.argv.pop(1)
+        sys.argv.remove('--insecure')
     else:
         insecure = False
     if len(sys.argv) > 1:
diff --git a/pyomo/core/base/external.py b/pyomo/core/base/external.py
--- a/pyomo/core/base/external.py
+++ b/pyomo/core/base/external.py
@@ -16,7 +16,7 @@
 
 from ctypes import (
     Structure, POINTER, CFUNCTYPE, cdll, byref,
-    c_int, c_long, c_double, c_byte, c_char_p, c_void_p )
+    c_int, c_long, c_ulong, c_double, c_byte, c_char_p, c_void_p )
 
 from pyomo.core.base.numvalue import as_numeric
 from pyomo.core.base.component import Component
@@ -106,14 +106,25 @@ def load_library(self):
                 self._so = cdll.LoadLibrary(self._library)
 
         self._known_functions = {}
+        AE = _AMPLEXPORTS()
+        AE.ASLdate = 20160307
         def addfunc(name, f, _type, nargs, funcinfo, ae):
             # trap for Python 3, where the name comes in as bytes() and
             # not a string
             if not isinstance(name, six.string_types):
                 name = name.decode()
             self._known_functions[str(name)] = (f, _type, nargs, funcinfo, ae)
-        AE = _AMPLEXPORTS()
         AE.Addfunc = _AMPLEXPORTS.ADDFUNC(addfunc)
+        def addrandinit(ae, rss, v):
+            # TODO: This should support the randinit ASL option
+            rss(v, 1)
+        AE.Addrandinit = _AMPLEXPORTS.ADDRANDINIT(addrandinit)
+        def atreset(ae, a, b):
+            logger.warning(
+                "AMPL External function: ignoring AtReset call in external "
+                "library.  This may result in a memory leak or other "
+                "undesirable behavior.")
+        AE.AtReset = _AMPLEXPORTS.ATRESET(atreset)
 
         FUNCADD = CFUNCTYPE( None, POINTER(_AMPLEXPORTS) )
         FUNCADD(('funcadd_ASL', self._so))(byref(AE))
@@ -248,6 +259,18 @@ class _AMPLEXPORTS(Structure):
         c_char_p, AMPLFUNC, c_int, c_int, c_void_p,
         POINTER(_AMPLEXPORTS) )
 
+    RANDSEEDSETTER = CFUNCTYPE(
+        None,
+        c_void_p, c_ulong )
+
+    ADDRANDINIT = CFUNCTYPE(
+        None,
+        POINTER(_AMPLEXPORTS), RANDSEEDSETTER, c_void_p )
+
+    ATRESET = CFUNCTYPE(
+        None,
+        POINTER(_AMPLEXPORTS), c_void_p, c_void_p )
+
     _fields_ = [
 	('StdErr', c_void_p),
 	('Addfunc', ADDFUNC),
@@ -261,7 +284,7 @@ class _AMPLEXPORTS(Structure):
 	('Crypto', c_void_p),
 	('asl', c_char_p),
 	('AtExit', c_void_p),
-	('AtReset', c_void_p),
+	('AtReset', ATRESET),
 	('Tempmem', c_void_p),
 	('Add_table_handler', c_void_p),
 	('Private', c_char_p),
@@ -308,5 +331,5 @@ class _AMPLEXPORTS(Structure):
 	('SnprintF', c_void_p),
 	('VsnprintF', c_void_p),
 	('Addrand', c_void_p),
-	('Addrandinit', c_void_p),
+	('Addrandinit', ADDRANDINIT),
 	]
diff --git a/pyomo/util/getGSL.py b/pyomo/util/getGSL.py
new file mode 100644
--- /dev/null
+++ b/pyomo/util/getGSL.py
@@ -0,0 +1,66 @@
+import zipfile
+import io
+import os
+import platform
+import ssl
+import stat
+import sys
+from six.moves.urllib.request import urlopen
+
+# These URLs were retrieved from
+#     https://ampl.com/resources/extended-function-library/
+urlmap = {
+    'linux':   'https://www.ampl.com/NEW/amplgsl/amplgsl.linux-intel%s.zip',
+    'windows': 'https://www.ampl.com/NEW/amplgsl/amplgsl.mswin%s.zip',
+    'cygwin':  'https://www.ampl.com/NEW/amplgsl/amplgsl.mswin%s.zip',
+    'darwin':  'https://www.ampl.com/NEW/amplgsl/amplgsl.macosx%s.zip'
+}
+
+def get_gsl(fname=None, insecure=False):
+    system = platform.system().lower()
+    for c in '.-_':
+        system = system.split(c)[0]
+    bits = 64 if sys.maxsize > 2**32 else 32
+    url = urlmap.get(system, None)
+    if url is None:
+        raise RuntimeError(
+            "ERROR: cannot infer the correct url for platform '%s'" % platform)
+    url = url % bits
+
+    if fname is None:
+        fname = '.'
+    if os.path.isdir(fname):
+        fname = os.path.join(fname, 'amplgsl.dll')
+
+    print("Fetching GSL from %s and installing it to %s" % (url, fname))
+    with open(fname, 'wb') as FILE:
+        try:
+            ctx = ssl.create_default_context()
+            if insecure:
+                ctx.check_hostname = False
+                ctx.verify_mode = ssl.CERT_NONE
+            fetch = urlopen(url, context=ctx)
+        except AttributeError:
+            # Revert to pre-2.7.9 syntax
+            fetch = urlopen(url)
+        zipped_file = io.BytesIO(fetch.read())
+        print("  ...downloaded %s bytes" % (len(zipped_file.getvalue()),))
+        raw_file = zipfile.ZipFile(zipped_file).open('amplgsl.dll').read()
+        FILE.write(raw_file)
+        print("  ...wrote %s bytes" % (len(raw_file),))
+
+if __name__ == '__main__':
+    if len(sys.argv) > 1 and '--insecure' in sys.argv:
+        insecure = True
+        sys.argv.remove('--insecure')
+    else:
+        insecure = False
+    if len(sys.argv) > 1:
+        fname = sys.argv.pop(1)
+    else:
+        fname = None
+    if len(sys.argv) > 1:
+        print("Usage: %s [--insecure] target" % sys.argv[0])
+        sys.exit(1)
+    get_gsl(fname=fname, insecure=insecure)
+


Test patch:
diff --git a/pyomo/core/tests/unit/test_external.py b/pyomo/core/tests/unit/test_external.py
--- a/pyomo/core/tests/unit/test_external.py
+++ b/pyomo/core/tests/unit/test_external.py
@@ -8,13 +8,14 @@
 #  This software is distributed under the 3-clause BSD License.
 #  ___________________________________________________________________________
 #
-
+import os
 import pyutilib.th as unittest
 
 from pyomo.core.base import IntegerSet
 from pyomo.environ import *
 from pyomo.core.base.external import (PythonCallbackFunction,
                                       AMPLExternalFunction)
+from pyomo.opt import check_available_solvers
 
 def _g(*args):
     return len(args)
@@ -74,5 +75,41 @@ def test_getname(self):
         self.assertEqual(M.m.f.getname(), "f")
         self.assertEqual(M.m.f.getname(True), "m.f")
 
+    def getGSL(self):
+        # Find the GSL DLL
+        DLL = None
+        for path in [os.getcwd()] + os.environ['PATH'].split(os.pathsep):
+            test = os.path.join(path, 'amplgsl.dll')
+            if os.path.isfile(test):
+                DLL = test
+                break
+        print("GSL found here: %s" % DLL)
+        return DLL
+
+    def test_eval_gsl_function(self):
+        DLL = self.getGSL()
+        if not DLL:
+            self.skipTest("Could not find the amplgsl.dll library in the PATH")
+        model = ConcreteModel()
+        model.z_func = ExternalFunction(library=DLL, function="gsl_sf_gamma")
+        model.x = Var(initialize=3, bounds=(1e-5,None))
+        model.o = Objective(expr=model.z_func(model.x))
+        self.assertAlmostEqual(value(model.o), 2.0, 7)
+
+    @unittest.skipIf(not check_available_solvers('ipopt'),
+                     "The 'ipopt' solver is not available")
+    def test_solve_gsl_function(self):
+        DLL = self.getGSL()
+        if not DLL:
+            self.skipTest("Could not find the amplgsl.dll library in the PATH")
+        model = ConcreteModel()
+        model.z_func = ExternalFunction(library=DLL, function="gsl_sf_gamma")
+        model.x = Var(initialize=3, bounds=(1e-5,None))
+        model.o = Objective(expr=model.z_func(model.x))
+        opt = SolverFactory('ipopt')
+        res = opt.solve(model, tee=True)
+        self.assertAlmostEqual(value(model.o), 0.885603194411, 7)
+
+
 if __name__ == "__main__":
     unittest.main()