Repository: Pyomo/pyomo

Issue:
Issues with NEOS solvers and solution output
I'm with the NEOS team and I'm trying to figure out some issues with Pyomo+NEOS.

It seems that 
`pyomo solve --solver=<solver> --solver-manager=neos model.py`
works only if `<solver>` has an explicit plugin (e.g., any of the solvers in pyomo/solvers/plugins/solvers).  This is mentioned in #910.  

The following seems to work for any NEOS-valid solver:
```
manager = SolverManagerFactory('neos')
results = manager.solve(model,opt='<solver>')
```

Is there a bug with the first method?  And if not, I assume that the second method is the way to go?  I dug around a little bit and it looks like the first method tries to create `SolverFactory(<solver>)` but this results in an unknown solver for some of the NEOS solvers.

Furthermore, in either case, no solution seems to be returned.  I don't see any error messages, but printing the results always shows 
```
Solution: 
- number of solutions: 0
  number of solutions displayed: 0
```
I'm not sure what Pyomo is expecting from NEOS, but if there's anything we can do on our end to fix this, let me know.



Discussion:
@gnowzil thanks for the note, it is certainly possible that this is a bug. We're hoping to rework Pyomo's solver interfaces this December which might fix this and leave us with a more robust interface to NEOS.
Hi,  I just wanted to follow up on this.  We've been getting some queries about pyomo on NEOS.  I ran small examples through all of the solvers available and was met with varying degrees of success.  I'm not an expert in using Pyomo so I may be doing this incorrectly, but I've summarized what I have seen below.

- cbc, conopt, cplex, mosek, and snopt appeared to work and returned a solution.
- filter sqp, knitro, l-bfgs-b, lancelot, lgo, loqo, minlp, minos, minto, ooqp appeared to work if I use the command `python myexample.py`, with `SolverManagerFactory('neos')` and the `solve` call in the script, but failed if I used `pyomo solve --solver-manager=neos --solver=<solver> mymodel.py`, where `mymodel.py` only has the model.  
- bonmin, couenne, and filmint didn't work at all.
- raposa is listed as a solver, but not recognized when solving with a `pyomo solve` call.  

Let me know if there's anything we can do to help.  One thing that could be causing problems is the return of solutions from NEOS -- we made some minor changes here and we may not be returning something that Pyomo expects.
Thanks for the information. We will discuss this at our next meeting...
Some further feedback on the issues that are noted above:

* I'm seeing NEOS queue failures for minto and raposa.  I'm not sure what is going on.  Perhaps NEOS is returning a *.ZIP file that we need to open up to get the *.sol file

* I added raposa to the documentation in Pyomo, so it's listed as a solver in my fork.  It seems to be launched.

* On a simple problem, the following solver generate the correct solution (using a script interface):
  * snopt, bonmin, cbc, conopt, couenne, cplex, filmint, filter, ipopt, knitro, lancelot, lgo, loqo, minlp, minos, mosek, snopt

* ooqp fails when I hand it a maximization problem, and solves the same problem when minimizing

* l-bfgs-b fails even on trivial problems (!)

Resolving patch:
diff --git a/pyomo/neos/__init__.py b/pyomo/neos/__init__.py
--- a/pyomo/neos/__init__.py
+++ b/pyomo/neos/__init__.py
@@ -31,6 +31,8 @@
 'mosek':    'Interior point NLP solver',
 'ooqp':     'Convex QP solver',
 'path':     'Nonlinear MCP solver',
-'snopt':    'SQP NLP solver'
+'snopt':    'SQP NLP solver',
+'raposa':   'A Global Solver for Polynomial Programming Problems',
+'lgo':      'Lipschitz-Continuous Global Optimizer'
 }
 
diff --git a/pyomo/neos/kestrel.py b/pyomo/neos/kestrel.py
--- a/pyomo/neos/kestrel.py
+++ b/pyomo/neos/kestrel.py
@@ -121,11 +121,25 @@ def make_connection(self, host):
         return ProxiedTransport_PY3()
 
 
-class kestrelAMPL:
+class kestrelAMPL(object):
 
     def __init__(self):
         self.setup_connection()
 
+    def __del__(self):
+        # This is a hack to force the socket to close.  When running
+        # tests under unittest, unittest will report any underlying
+        # socket resources that have not been closed as warnings.  This
+        # will cause the socket to be closed when the kestrelAMPL object
+        # falls out of scope and will suppress the warnings from
+        # unittest.
+        #
+        # Note that this is only to suppress warnings, as __del__ is not
+        # guaranteed to be called (especially for any objects that still
+        # exist when the Python process terminates)
+        if self.neos is not None:
+            self.transport.close()
+
     def setup_connection(self):
         # on *NIX, the proxy can show up either upper or lowercase.
         # Prefer lower case, and prefer HTTPS over HTTP if the
@@ -137,14 +151,17 @@ def setup_connection(self):
             proxy = os.environ.get(
                 'https_proxy', os.environ.get(
                     'HTTPS_PROXY', proxy))
-        transport = None
         if proxy:
-            transport = ProxiedTransport()
-            transport.set_proxy(proxy)
+            self.transport = ProxiedTransport()
+            self.transport.set_proxy(proxy)
+        elif NEOS.scheme == 'https':
+            self.transport = xmlrpclib.SafeTransport()
+        else:
+            self.transport = xmlrpclib.Transport()
 
         self.neos = xmlrpclib.ServerProxy(
             "%s://%s:%s" % (NEOS.scheme, NEOS.host, NEOS.port),
-            transport=transport)
+            transport=self.transport)
 
         logger.info("Connecting to the NEOS server ... ")
         try:
@@ -181,9 +198,8 @@ def retrieve(self,stub,jobNumber,password):
         results = self.neos.getFinalResults(jobNumber,password)
         if isinstance(results,xmlrpclib.Binary):
             results = results.data
-        #decode results to kestrel.sol
-        # Well try to anyway, any errors will result in error strings in .sol file
-        #  instead of solution.
+        # decode results to kestrel.sol; well try to anyway, any errors
+        # will result in error strings in .sol file instead of solution.
         if stub[-4:] == '.sol':
             stub = stub[:-4]
         solfile = open(stub + ".sol","wb")
diff --git a/pyomo/scripting/plugins/solve.py b/pyomo/scripting/plugins/solve.py
--- a/pyomo/scripting/plugins/solve.py
+++ b/pyomo/scripting/plugins/solve.py
@@ -12,7 +12,7 @@
 import sys
 import argparse
 
-from pyomo.opt import SolverFactory
+from pyomo.opt import SolverFactory, UnknownSolver
 from pyomo.scripting.pyomo_parser import add_subparser, CustomHelpFormatter
 
 
@@ -29,6 +29,10 @@ def create_parser(parser=None):
         action='store',
         dest='solver',
         default=None)
+    parser.add_argument('--solver-manager',
+        action='store',
+        dest='solver_manager',
+        default='serial')
     parser.add_argument('--generate-config-template',
         action='store',
         dest='template',
@@ -126,9 +130,12 @@ def create_temporary_parser(solver=False, generate=False):
 
 
 def solve_exec(args, unparsed):
+
     import pyomo.scripting.util
     #
+    solver_manager = getattr(args,'solver_manager',None)
     solver = getattr(args,'solver',None)
+    #
     if solver is None:
         #
         # Get configuration values if no solver has been specified
@@ -159,8 +166,9 @@ def solve_exec(args, unparsed):
             sys.exit(1)
 
     config = None
-    with SolverFactory(solver) as opt:
-        if opt is None:
+    _solver = solver if solver_manager == 'serial' else 'asl'
+    with SolverFactory(_solver) as opt:
+        if opt is None or isinstance(opt, UnknownSolver):
             print("ERROR: Unknown solver '%s'!" % solver)
             sys.exit(1)
         #
@@ -202,6 +210,8 @@ def solve_exec(args, unparsed):
 
     if config is None:
         raise RuntimeError("Failed to create config object")
+    config.solvers[0].solver_name = solver
+    config.solvers[0].manager = solver_manager
 
     from pyomo.scripting.pyomo_command import run_pyomo
     #


Test patch:
diff --git a/.github/workflows/pr_master_test.yml b/.github/workflows/pr_master_test.yml
--- a/.github/workflows/pr_master_test.yml
+++ b/.github/workflows/pr_master_test.yml
@@ -94,6 +94,14 @@ jobs:
           TARGET: linux
           PYENV: pip
 
+        - os: ubuntu-18.04
+          python: 3.7
+          other: /singletest
+          category: neos
+          skip_doctest: 1
+          TARGET: linux
+          PYENV: pip
+
         - os: ubuntu-18.04
           python: 3.6
           other: /cython
@@ -506,8 +514,11 @@ jobs:
     - name: Run Pyomo tests
       if: matrix.mpi == 0
       run: |
-        test.pyomo -v --cat="${{matrix.category}}" \
-            pyomo `pwd`/pyomo-model-libraries
+        CATEGORY=
+        for cat in ${{matrix.category}}; do
+            CATEGORY+=" --cat $cat"
+        done
+        test.pyomo -v $CATEGORY pyomo `pwd`/pyomo-model-libraries
 
     - name: Run Pyomo MPI tests
       if: matrix.mpi != 0
diff --git a/.github/workflows/push_branch_test.yml b/.github/workflows/push_branch_test.yml
--- a/.github/workflows/push_branch_test.yml
+++ b/.github/workflows/push_branch_test.yml
@@ -505,8 +505,11 @@ jobs:
     - name: Run Pyomo tests
       if: matrix.mpi == 0
       run: |
-        test.pyomo -v --cat="${{matrix.category}}" \
-            pyomo `pwd`/pyomo-model-libraries
+        CATEGORY=
+        for cat in ${{matrix.category}}; do
+            CATEGORY+=" --cat $cat"
+        done
+        test.pyomo -v $CATEGORY pyomo `pwd`/pyomo-model-libraries
 
     - name: Run Pyomo MPI tests
       if: matrix.mpi != 0
diff --git a/pyomo/neos/tests/t1.py b/pyomo/neos/tests/model_min_lp.py
similarity index 60%
rename from pyomo/neos/tests/t1.py
rename to pyomo/neos/tests/model_min_lp.py
--- a/pyomo/neos/tests/t1.py
+++ b/pyomo/neos/tests/model_min_lp.py
@@ -8,12 +8,17 @@
 #  This software is distributed under the 3-clause BSD License.
 #  ___________________________________________________________________________
 
-from pyomo.environ import ConcreteModel, Var, Objective, Constraint, maximize
+import pyomo.environ as pyo
 
-M = ConcreteModel()
-M.x = Var(bounds=(0,1))
-M.o = Objective(expr=2*M.x, sense=maximize)
-M.c = Constraint(expr=M.x <= 0.5)
+model = pyo.ConcreteModel()
+model.y = pyo.Var(bounds=(-10,10), initialize=0.5)
+model.x = pyo.Var(bounds=(-5,5), initialize=0.5)
 
-model = M
+@model.ConstraintList()
+def c(m):
+    yield m.y >= m.x - 2
+    yield m.y >= - m.x
+    yield m.y <= m.x
+    yield m.y <= 2 - m.x
 
+model.obj = pyo.Objective(expr=model.y, sense=pyo.minimize)
diff --git a/pyomo/neos/tests/t1.txt b/pyomo/neos/tests/t1.txt
deleted file mode 100644
--- a/pyomo/neos/tests/t1.txt
+++ /dev/null
@@ -1,37 +0,0 @@
-# ==========================================================
-# = Solver Results                                         =
-# ==========================================================
-# ----------------------------------------------------------
-#   Problem Information
-# ----------------------------------------------------------
-Problem: 
-- Lower bound: -inf
-  Upper bound: inf
-  Number of objectives: 1
-  Number of constraints: 1
-  Number of variables: 1
-  Sense: unknown
-# ----------------------------------------------------------
-#   Solver Information
-# ----------------------------------------------------------
-Solver: 
-- Status: ok
-  Message: CBC trunk optimal, objective -1; 0 iterations
-  Termination condition: optimal
-  Id: 0
-# ----------------------------------------------------------
-#   Solution Information
-# ----------------------------------------------------------
-Solution: 
-- number of solutions: 1
-  number of solutions displayed: 1
-- Status: optimal
-  Objective: 
-    o: 
-      Id: 0
-      Value: 1.0
-  Variable: 
-    x:
-      Id: 0
-      Value: 0.5
-  Constraint: No nonzero values
diff --git a/pyomo/neos/tests/test_neos.py b/pyomo/neos/tests/test_neos.py
--- a/pyomo/neos/tests/test_neos.py
+++ b/pyomo/neos/tests/test_neos.py
@@ -10,24 +10,30 @@
 #
 # Test NEOS solver interface
 #
+# Because the Kestrel tests require connections to the NEOS server, and
+# that can take quite a while (5-20+ seconds), we will only run these
+# tests as part of the nightly suite (i.e., by the CI system as part of
+# PR / master tests)
+#
 
 import os
-from os.path import abspath, dirname
-currdir = dirname(abspath(__file__))
+import json
+import os.path
 
 import pyutilib.th as unittest
 
-from pyomo.common.dependencies import yaml, yaml_available, yaml_load_args
 import pyomo.scripting.pyomo_command as main
 from pyomo.scripting.util import cleanup
 from pyomo.neos.kestrel import kestrelAMPL
+import pyomo.neos
 
 import pyomo.environ as pyo
 
+from pyomo.common.fileutils import this_file_dir
+currdir = this_file_dir()
+
 neos_available = False
 try:
-    # Attempt a connection to NEOS.  Any failure will result in skipping
-    # these tests
     if kestrelAMPL().neos is not None:
         neos_available = True
 except:
@@ -38,58 +44,228 @@
     email_set = False
 
 
-#
-# Because the Kestrel tests require connections to the NEOS server, and
-# that can take quite a while (5-20+ seconds), we will only run these
-# tests as part of the nightly suite (i.e., by the CI system as part of
-# PR / master tests)
-#
-@unittest.category('nightly')
+def _model(sense):
+    # Goals of this model:
+    # - linear
+    # - solution has nonzero variable values (so they appear in the results)
+    model = pyo.ConcreteModel()
+    model.y = pyo.Var(bounds=(-10,10), initialize=0.5)
+    model.x = pyo.Var(bounds=(-5,5), initialize=0.5)
+
+    @model.ConstraintList()
+    def c(m):
+        yield m.y >= m.x - 2
+        yield m.y >= - m.x
+        yield m.y <= m.x
+        yield m.y <= 2 - m.x
+
+    model.obj = pyo.Objective(expr=model.y, sense=sense)
+    return model
+
+
+@unittest.category('nightly', 'neos')
 @unittest.skipIf(not neos_available, "Cannot make connection to NEOS server")
-@unittest.skipIf(not email_set, "NEOS_EMAIL not set")
+@unittest.skipUnless(email_set, "NEOS_EMAIL not set")
 class TestKestrel(unittest.TestCase):
 
-    @unittest.skipIf(not yaml_available, "YAML is not available")
-    def test_pyomo_command(self):
-        results = os.path.join(currdir, 'result.yml')
+    def test_doc(self):
+        kestrel = kestrelAMPL()
+        tmp = [tuple(name.split(':')) for name in kestrel.solvers()]
+        amplsolvers = set(v[0].lower() for v in tmp if v[1]=='AMPL')
+
+        doc = pyomo.neos.doc
+        dockeys = set(doc.keys())
+
+        self.assertEqual(amplsolvers, dockeys)
+
+        #gamssolvers = set(v[0].lower() for v in tmp if v[1]=='GAMS')
+        #missing = gamssolvers - amplsolvers
+        #self.assertEqual(len(missing) == 0)
+
+
+class RunAllNEOSSolvers(object):
+    def test_bonmin(self):
+        self._run('bonmin')
+
+    def test_cbc(self):
+        self._run('cbc')
+
+    def test_conopt(self):
+        self._run('conopt')
+
+    def test_couenne(self):
+        self._run('couenne')
+
+    def test_cplex(self):
+        self._run('cplex')
+
+    def test_filmint(self):
+        self._run('filmint')
+
+    def test_filter(self):
+        self._run('filter')
+
+    def test_ipopt(self):
+        self._run('ipopt')
+
+    def test_knitro(self):
+        self._run('knitro')
+
+    # This solver only handles bound constrained variables
+    def test_lbfgsb(self):
+        self._run('l-bfgs-b', False)
+
+    def test_lancelot(self):
+        self._run('lancelot')
+
+    def test_loqo(self):
+        self._run('loqo')
+
+    def test_minlp(self):
+        self._run('minlp')
+
+    def test_minos(self):
+        self._run('minos')
+
+    def test_minto(self):
+        self._run('minto')
+
+    def test_mosek(self):
+        self._run('mosek')
+
+    def test_ooqp(self):
+        if self.sense == pyo.maximize:
+            # OOQP does not recognize maximization problems and
+            # minimizes instead.
+            with self.assertRaisesRegex(
+                    AssertionError, '.* != 1 within'):
+                self._run('ooqp')
+        else:
+            self._run('ooqp')
+
+    # The simple tests aren't complementarity 
+    # problems
+    #def test_path(self):
+    #    self._run('path')
+
+    def test_snopt(self):
+        self._run('snopt')
+
+    def test_raposa(self):
+        # RAPOSa does not appear to recognize maximization problems and
+        # returns an incorrect result (as of 29 Jan 20: 2.5)
+        if self.sense == pyo.maximize:
+            with self.assertRaisesRegex(
+                    AssertionError, '.* != 1 within'):
+                self._run('raposa')
+        else:
+            self._run('raposa')
+
+    def test_lgo(self):
+        self._run('lgo')
+
+
+class DirectDriver(object):
+    def _run(self, opt, constrained=True):
+        m = _model(self.sense)
+        solver_manager = pyo.SolverManagerFactory('neos')
+        results = solver_manager.solve(m, opt=opt)
+
+        expected_y = {
+            (pyo.minimize, True): -1,
+            (pyo.maximize, True):  1,
+            (pyo.minimize, False): -10,
+            (pyo.maximize, False): 10,
+        }[self.sense, constrained]
+
+        self.assertEqual(results.solver[0].status, pyo.SolverStatus.ok)
+        if constrained:
+            # If the solver ignores constraints, x is degenerate
+            self.assertAlmostEqual(pyo.value(m.x), 1, delta=1e-5)
+        self.assertAlmostEqual(pyo.value(m.obj), expected_y, delta=1e-5)
+        self.assertAlmostEqual(pyo.value(m.y), expected_y, delta=1e-5)
+
+class PyomoCommandDriver(object):
+
+    def _run(self, opt, constrained=True):
+        expected_y = {
+            (pyo.minimize, True): -1,
+            (pyo.maximize, True):  1,
+            (pyo.minimize, False): -10,
+            (pyo.maximize, False): 10,
+        }[self.sense, constrained]
+
+        filename = 'model_min_lp.py' if self.sense == pyo.minimize \
+                   else 'model_max_lp.py'
+
+        results = os.path.join(currdir, 'result.json')
         args = [
-            os.path.join(currdir,'t1.py'),
+            os.path.join(currdir, filename),
             '--solver-manager=neos',
-            '--solver=cbc',
-            '--symbolic-solver-labels',
+            '--solver=%s' % opt,
+            '--logging=quiet',
             '--save-results=%s' % results,
-            '-c',
+            '--results-format=json',
+            '-c'
             ]
         try:
             output = main.run(args)
             self.assertEqual(output.errorcode, 0)
 
             with open(results) as FILE:
-                data = yaml.load(FILE, **yaml_load_args)
-            self.assertEqual(
-                data['Solver'][0]['Status'], 'ok')
-            self.assertAlmostEqual(
-                data['Solution'][1]['Status'], 'optimal')
-            self.assertAlmostEqual(
-                data['Solution'][1]['Objective']['o']['Value'], 1)
-            self.assertAlmostEqual(
-                data['Solution'][1]['Variable']['x']['Value'], 0.5)
+                data = json.load(FILE)
         finally:
             cleanup()
-            os.remove(results)
+            if os.path.exists(results):
+                os.remove(results)
 
-    def test_kestrel_plugin(self):
-        m = pyo.ConcreteModel()
-        m.x = pyo.Var(bounds=(0,1), initialize=0)
-        m.c = pyo.Constraint(expr=m.x <= 0.5)
-        m.obj = pyo.Objective(expr=2*m.x, sense=pyo.maximize)
+        self.assertEqual(
+            data['Solver'][0]['Status'], 'ok')
+        self.assertEqual(
+            data['Solution'][1]['Status'], 'optimal')
+        self.assertAlmostEqual(
+            data['Solution'][1]['Objective']['obj']['Value'],
+            expected_y, delta=1e-5)
+        if constrained:
+            # If the solver ignores constraints, x is degenerate
+            self.assertAlmostEqual(
+                data['Solution'][1]['Variable']['x']['Value'],
+                1, delta=1e-5)
+        self.assertAlmostEqual(
+            data['Solution'][1]['Variable']['y']['Value'],
+            expected_y, delta=1e-5)
 
-        solver_manager = pyo.SolverManagerFactory('neos')
-        results = solver_manager.solve(m, opt='cbc')
 
-        self.assertEqual(results.solver[0].status, pyo.SolverStatus.ok)
-        self.assertAlmostEqual(pyo.value(m.x), 0.5)
+@unittest.category('neos')
+@unittest.skipUnless(email_set, "NEOS_EMAIL not set")
+@unittest.skipIf(not neos_available, "Cannot make connection to NEOS server")
+class TestSolvers_direct_call_min(RunAllNEOSSolvers, DirectDriver,
+                                  unittest.TestCase):
+    sense = pyo.minimize
+
+    # Add the CBC test to the nightly suite
+    @unittest.category('nightly')
+    def test_cbc(self):
+        super(TestSolvers_direct_call_min, self).test_cbc()
+
+@unittest.category('neos')
+@unittest.skipUnless(email_set, "NEOS_EMAIL not set")
+@unittest.skipIf(not neos_available, "Cannot make connection to NEOS server")
+class TestSolvers_direct_call_max(RunAllNEOSSolvers, DirectDriver,
+                                  unittest.TestCase):
+    sense = pyo.maximize
+
+@unittest.category('neos')
+@unittest.skipUnless(email_set, "NEOS_EMAIL not set")
+@unittest.skipIf(not neos_available, "Cannot make connection to NEOS server")
+class TestSolvers_pyomo_cmd_min(RunAllNEOSSolvers, PyomoCommandDriver,
+                                unittest.TestCase):
+    sense = pyo.minimize
 
+    # Add the CBC test to the nightly suite
+    @unittest.category('nightly')
+    def test_cbc(self):
+        super(TestSolvers_pyomo_cmd_min, self).test_cbc()
 
 if __name__ == "__main__":
     unittest.main()