Repository: Pyomo/pyomo

Issue:
AMPLFUNC Environment Variable
It seems that when you call an ASL solver multiple times from a Python script, and you use ExternalFucntion, the AMPLFUNC environment variable accumulates a bunch of repeated copies of the .so file.  I haven't determined if this causes a real problem yet, but it causes the ASL to spew a bunch of warnings about already defined functions.  I printed the environment variables during an initialization consisting of several solves, so you can get an idea of what's happening

'PYOMO_AMPLFUNC': '../../../properties/physical/MEA_Simple/phys_prop.so'

'AMPLFUNC': '/imports/users/jeslick/git/eslickj/models/idaes_models/properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so'
AMPLFUNC Environment Variable
It seems that when you call an ASL solver multiple times from a Python script, and you use ExternalFucntion, the AMPLFUNC environment variable accumulates a bunch of repeated copies of the .so file.  I haven't determined if this causes a real problem yet, but it causes the ASL to spew a bunch of warnings about already defined functions.  I printed the environment variables during an initialization consisting of several solves, so you can get an idea of what's happening

'PYOMO_AMPLFUNC': '../../../properties/physical/MEA_Simple/phys_prop.so'

'AMPLFUNC': '/imports/users/jeslick/git/eslickj/models/idaes_models/properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so\n../../../properties/physical/MEA_Simple/phys_prop.so'


Discussion:
Can you add a ticket for this?  This looks like a bug.

On Mon, Nov 21, 2016 at 1:38 PM, John Eslick notifications@github.com
wrote:

> It seems that when you call an ASL solver multiple times from a Python
> script, and you use ExternalFucntion, the AMPLFUNC environment variable
> accumulates a bunch of repeated copies of the .so file. I haven't
> determined if this causes a real problem yet, but it causes the ASL to spew
> a bunch of warnings about already defined functions. I printed the
> environment variables during an initialization consisting of several
> solves, so you can get an idea of what's happening
> 
> 'PYOMO_AMPLFUNC': '../../../properties/physical/MEA_Simple/phys_prop.so'
> 
> 'AMPLFUNC': '/imports/users/jeslick/git/eslickj/models/idaes_models/
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so'
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Pyomo/pyomo/issues/73, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AAsb-KXaYwWeP_eYDg7r0PRdpIidtggNks5rAeUXgaJpZM4K4jAD
> .

Sorry, I'm pretty new to how github works.  I made an issue.  How do I go about adding a ticket?
Ah.  I missed that this email was generated from an issue.  Thanks!

On Mon, Nov 21, 2016 at 1:47 PM, John Eslick notifications@github.com
wrote:

> Sorry, I'm pretty new to how github works. I made an issue. How do I go
> about adding a ticket?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/Pyomo/pyomo/issues/73#issuecomment-262029779, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AAsb-LWzBGWcVWVJij7vWZ0yFp4gBWIRks5rAedFgaJpZM4K4jAD
> .

Thanks for looking into it.  I've been having some trouble with the solver failing mysteriously, and after a lot of searching this seems to be the culprit.  I did some more testing, and after a certain number of times importing the .so file the ipopt solver segfaults before even getting started.  I have no idea why that would happen. It may be something weird in my user function code.  Anyway this bug does seem to potentially cause more problems than just annoying warning messages. 
To follow up on this issue, a couple other people working on IDAES have had problems with ipopt crashing when a library gets included in AMPLFUNC too many times, so this does seem to be an important issue.  Previously I wasn't totally sure whether or not this was a serious issue.
@eslickj: can you take a look at Pyomo/IDAES-premerge (240d3f9) and see if that resolves this issue?
@eslickj Has this been resolved?
I don't think so.  Just pulled the latest from Master. @jsiirola sorry I probably missed when you asked me to test this back in June.

Here is a bit of code I used to test.  Works if I set AMPLFUNC myself, but not if I don't. Not sure if you have a handy test for this, but I can get you the external function code if it helps.
```Python
from pyomo.environ import *
from pyomo.opt import SolverFactory
import os 

if __name__ == "__main__":
    model = ConcreteModel()
    flib = "./phys_prop.so"
    model.z_func = ExternalFunction(library=flib, function="z_func")
    model.x = Var(initialize=3)
    model.o = Objective(expr=model.z_func(model.x))
    opt = SolverFactory('ipopt')
    os.environ["AMPLFUNC"] = flib
    res = opt.solve(model)
    display(model)
```
Can you add a ticket for this?  This looks like a bug.

On Mon, Nov 21, 2016 at 1:38 PM, John Eslick notifications@github.com
wrote:

> It seems that when you call an ASL solver multiple times from a Python
> script, and you use ExternalFucntion, the AMPLFUNC environment variable
> accumulates a bunch of repeated copies of the .so file. I haven't
> determined if this causes a real problem yet, but it causes the ASL to spew
> a bunch of warnings about already defined functions. I printed the
> environment variables during an initialization consisting of several
> solves, so you can get an idea of what's happening
> 
> 'PYOMO_AMPLFUNC': '../../../properties/physical/MEA_Simple/phys_prop.so'
> 
> 'AMPLFUNC': '/imports/users/jeslick/git/eslickj/models/idaes_models/
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so\n../../../
> properties/physical/MEA_Simple/phys_prop.so'
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Pyomo/pyomo/issues/73, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AAsb-KXaYwWeP_eYDg7r0PRdpIidtggNks5rAeUXgaJpZM4K4jAD
> .

Sorry, I'm pretty new to how github works.  I made an issue.  How do I go about adding a ticket?
Ah.  I missed that this email was generated from an issue.  Thanks!

On Mon, Nov 21, 2016 at 1:47 PM, John Eslick notifications@github.com
wrote:

> Sorry, I'm pretty new to how github works. I made an issue. How do I go
> about adding a ticket?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/Pyomo/pyomo/issues/73#issuecomment-262029779, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AAsb-LWzBGWcVWVJij7vWZ0yFp4gBWIRks5rAedFgaJpZM4K4jAD
> .

Thanks for looking into it.  I've been having some trouble with the solver failing mysteriously, and after a lot of searching this seems to be the culprit.  I did some more testing, and after a certain number of times importing the .so file the ipopt solver segfaults before even getting started.  I have no idea why that would happen. It may be something weird in my user function code.  Anyway this bug does seem to potentially cause more problems than just annoying warning messages. 
To follow up on this issue, a couple other people working on IDAES have had problems with ipopt crashing when a library gets included in AMPLFUNC too many times, so this does seem to be an important issue.  Previously I wasn't totally sure whether or not this was a serious issue.
@eslickj: can you take a look at Pyomo/IDAES-premerge (240d3f9) and see if that resolves this issue?
@eslickj Has this been resolved?
I don't think so.  Just pulled the latest from Master. @jsiirola sorry I probably missed when you asked me to test this back in June.

Here is a bit of code I used to test.  Works if I set AMPLFUNC myself, but not if I don't. Not sure if you have a handy test for this, but I can get you the external function code if it helps.
```Python
from pyomo.environ import *
from pyomo.opt import SolverFactory
import os 

if __name__ == "__main__":
    model = ConcreteModel()
    flib = "./phys_prop.so"
    model.z_func = ExternalFunction(library=flib, function="z_func")
    model.x = Var(initialize=3)
    model.o = Objective(expr=model.z_func(model.x))
    opt = SolverFactory('ipopt')
    os.environ["AMPLFUNC"] = flib
    res = opt.solve(model)
    display(model)
```

Resolving patch:
diff --git a/pyomo/solvers/plugins/solvers/CONOPT.py b/pyomo/solvers/plugins/solvers/CONOPT.py
--- a/pyomo/solvers/plugins/solvers/CONOPT.py
+++ b/pyomo/solvers/plugins/solvers/CONOPT.py
@@ -110,6 +110,16 @@ def create_command_line(self, executable, problem_files):
         # Define command line
         #
         env=os.environ.copy()
+        #
+        # Merge the PYOMO_AMPLFUNC (externals defined within
+        # Pyomo/Pyomo) with any user-specified external function
+        # libraries
+        #
+        if 'PYOMO_AMPLFUNC' in env:
+            if 'AMPLFUNC' in env:
+                env['AMPLFUNC'] += "\n" + env['PYOMO_AMPLFUNC']
+            else:
+                env['AMPLFUNC'] = env['PYOMO_AMPLFUNC']
 
         cmd = [executable, problem_files[0], '-AMPL']
         if self._timer:
diff --git a/pyomo/solvers/plugins/solvers/IPOPT.py b/pyomo/solvers/plugins/solvers/IPOPT.py
--- a/pyomo/solvers/plugins/solvers/IPOPT.py
+++ b/pyomo/solvers/plugins/solvers/IPOPT.py
@@ -112,6 +112,16 @@ def create_command_line(self, executable, problem_files):
         # Define command line
         #
         env=os.environ.copy()
+        #
+        # Merge the PYOMO_AMPLFUNC (externals defined within
+        # Pyomo/Pyomo) with any user-specified external function
+        # libraries
+        #
+        if 'PYOMO_AMPLFUNC' in env:
+            if 'AMPLFUNC' in env:
+                env['AMPLFUNC'] += "\n" + env['PYOMO_AMPLFUNC']
+            else:
+                env['AMPLFUNC'] = env['PYOMO_AMPLFUNC']
 
         cmd = [executable, problem_files[0], '-AMPL']
         if self._timer:
diff --git a/pyomo/solvers/plugins/solvers/SCIPAMPL.py b/pyomo/solvers/plugins/solvers/SCIPAMPL.py
--- a/pyomo/solvers/plugins/solvers/SCIPAMPL.py
+++ b/pyomo/solvers/plugins/solvers/SCIPAMPL.py
@@ -109,6 +109,16 @@ def create_command_line(self, executable, problem_files):
         # Define command line
         #
         env=os.environ.copy()
+        #
+        # Merge the PYOMO_AMPLFUNC (externals defined within
+        # Pyomo/Pyomo) with any user-specified external function
+        # libraries
+        #
+        if 'PYOMO_AMPLFUNC' in env:
+            if 'AMPLFUNC' in env:
+                env['AMPLFUNC'] += "\n" + env['PYOMO_AMPLFUNC']
+            else:
+                env['AMPLFUNC'] = env['PYOMO_AMPLFUNC']
 
         cmd = [executable, problem_files[0], '-AMPL']
         if self._timer:


Test patch: