Repository: Pyomo/pyomo

Issue:
GAMS put file output with GAMS version >= 39
## Summary

GAMS 39 has a breaking update to the put file statements. Put statements of the form 
https://github.com/Pyomo/pyomo/blob/6fd5b8aa94b21bfe3290bdb019b33b822d51b681/pyomo/repn/plugins/gams_writer.py#L816
are faulty (now) as numbers don't get printed with a leading space. Therefore you can get space-less lines, thus your parsing of those files fails. You can fix by adding spaces in between values.

Same in:
https://github.com/Pyomo/pyomo/blob/6fd5b8aa94b21bfe3290bdb019b33b822d51b681/pyomo/repn/plugins/gams_writer.py#L818
https://github.com/Pyomo/pyomo/blob/6fd5b8aa94b21bfe3290bdb019b33b822d51b681/pyomo/repn/plugins/gams_writer.py#L829

### Steps to reproduce the issue

Not sure, was brought to our attention through GAMS support.

### Error Message

Not sure, was brought to our attention through GAMS support.

### Information on your system

Pyomo version: latest
Python version:
Operating system:
How Pyomo was installed (PyPI, conda, source):
Solver (if applicable): GAMS 39


### Additional information




Discussion:
Thanks, @renkekuhlmann - we'll look into this and work on adding the newest GAMS version to our testing infrastructure.
@MAZamarripa reported this in #2446 - it has the exact error message.

## Summary
Pyomo GAMS writer is failing to convert IDAES models to GAMS code. 
potential issues: expressions are not included in the GAMS writer (example below "func_tau" showing error $140 "unknown symbol"

I get these errors:
      34  c1.. x1 - ((647.096/func_tau((222.03374024716794), (101.325))) -
      (647.096/func_tau((0.05550843506179198*x2), (0.001*x3)))) =e= 0 ;
    ****                             $140,145              $8,37,409 ****   8
    ')' expected ****  37  '=l=' or '=e=' or '=g=' operator expected **** 140
    Unknown symbol **** 145  Set identifier or quoted element expected ****
    409  Unrecognizable item - skip to find a new statement ****
    looking for a ';' or a key word to get started again
      35  c2.. x4 - ((647.096/func_tau((0.05550843506179198*x5), (0.001*x6)))
      - (647.096/func_tau((194.27952271627194), (101.325)))) =e= 0 ;
    ****                               $145                $8  $8,37,409 ****
    8  ')' expected ****  37  '=l=' or '=e=' or '=g=' operator expected ****
    145  Set identifier or quoted element expected **** 409  Unrecognizable
    item - skip to find a new statement ****        looking for a ';' or a key
    word to get started again
      44  c11.. GAMS_OBJECTIVE =e= x11  ==  1 ;
    ****                                 $119 **** 119  Number (primary)
    expected
      77  SOLVE GAMS_MODEL USING nlp minimizing GAMS_OBJECTIVE;
    ****                                                      $257 **** 257
    Solve statement not checked because of previous errors
     119  put c1 c1.l c1.m /;
    ****            $141 **** 141  Symbol declared but no values have been
    assigned. Check for missing ****         data definition, assignment, data
    loading or implicit assignment ****         via a solve statement. ****
    A wild shot: You may have spurious commas in the explanatory ****
    text of a declaration. Check symbol reference list.
     120  put c2 c2.l c2.m /;

### Steps to reproduce the issue

<!-- Please include the command run and/or provide an example script that demonstrates the problem. -->
run example attached (run_heat_exchanger.py)


### Error Message

<!-- If Pyomo reported an error, provide the error message. PLEASE INCLUDE THE ENTIRE STACK TRACE. -->

<!-- If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
```console
$ # Output message here, including entire stack trace, if available
```
Traceback (most recent call last):
  File ".\run_heat_exchanger.py", line 157, in <module>
    add_options=['gams_model.optfile=1;'])
  File "C:\Users\ZAMARRIM\.conda\envs\idaes\lib\site-packages\pyomo\solvers\plugins\solvers\GAMS.py", line 845, in solve
    raise RuntimeError("GAMS encountered an error during solve. "
RuntimeError: GAMS encountered an error during solve. Check listing file for details.
### Information on your system

Pyomo version: 6.4.0
Python version: 3.7.13
Operating system: Windows 10
How Pyomo was installed (PyPI, conda, source): via idaes extensions (I will guess via conda)

Solver (if applicable): GAMS
Release: GAMS Studio 1.10.2 64 bit
Build Date: May 13 2022 14:35:10
GAMS Release: 39.1.1 dbf03b78 WEX-WEI x86 64bit/MS Windows
Release Date: 2022         

### Additional information

<!-- If you have any additional information, please list it here. -->
the attachment includes an idaes model that can be used to replicate the issue.
[run_heat_exchanger.zip](https://github.com/Pyomo/pyomo/files/8976703/run_heat_exchanger.zip)

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question (how to use Pyomo, what it can do, etc.), please refer to our [online documentation](https://github.com/Pyomo/pyomo#getting-help) on ways to get help. -->

There are other issues that I observed during testing but cannot replicate with the small example. 

**Issue with Baron solver:**  failing to run Baron solver (see below) 
 For information on BARON, see https://minlp.com/about-baron
ERROR: Solver (baron) returned non-zero return code (3)
ERROR: See the solver log above for diagnostic information.
Traceback (most recent call last):
  File "c:/Users/ZAMARRIM/Documents/IDAES2/idaes_examples_run/run_heat_exchanger.py", line 164, in <module>
    results = solver.solve(m, tee=True)
  File "C:\Users\ZAMARRIM\.conda\envs\idaes\lib\site-packages\pyomo\opt\base\solvers.py", line 596, in solve
    "Solver (%s) did not exit normally" % self.name)
pyomo.common.errors.ApplicationError: Solver (baron) did not exit normally


**Issue with pyomo model:** The pyomo model is running in GAMS (solving to optimal) but failing to return variable values from GAMS to Pyomo

--- DICOPT: Best integer solution found: 367758.749140
--- Reading solution for model GAMS_MODEL
--- Executing after solve: elapsed 0:00:25.021
--- model.gms(277999) 90 Mb
--- model.gms(287562) 90 Mb
--- model.gms(403886) 90 Mb
--- model.gms(417031) 90 Mb
--- Putfile results C:\Users\ZAMARRIM\Documents\IDAES2\idaes_examples_run\temp\results.dat
--- Putfile statresults C:\Users\ZAMARRIM\Documents\IDAES2\idaes_examples_run\temp\resultsstat.dat
*** Status: Normal completion
--- model.gms(417031) 86 Mb
--- Job model.gms Stop 06/24/22 10:03:30 elapsed 0:00:25.265

GAMS WORKING DIRECTORY: temp

Traceback (most recent call last):
  File "c:/Users/ZAMARRIM/Documents/Github/project-pareto/pareto/strategic_water_management/run_strategic_model.py", line 139, in <module>
    solve_model(model=strategic_model, options=options)
  File "c:\users\zamarrim\documents\github\project-pareto\pareto\strategic_water_management\strategic_produced_water_optimization.py", line 7509, in solve_model
    solve_MINLP_quality(model, opt)
  File "c:\users\zamarrim\documents\github\project-pareto\pareto\strategic_water_management\strategic_produced_water_optimization.py", line 7126, in solve_MINLP_quality
    results = SolverFactory(solver_source).solve(
  File "C:\Users\ZAMARRIM\.conda\envs\pareto\lib\site-packages\pyomo\solvers\plugins\solvers\GAMS.py", line 852, in solve
    model_soln, stat_vars = self._parse_dat_results(
  File "C:\Users\ZAMARRIM\.conda\envs\pareto\lib\site-packages\pyomo\solvers\plugins\solvers\GAMS.py", line 1222, in _parse_dat_results
    **stat_vars[items[0]] = float(items[1])
IndexError: list index out of range**
(idaes) PS C:\Users\ZAMARRIM\Documents\IDA
As a note - the GAMS solver interface doesn't always print the most helpful messages. I had forgotten to put the license in the main GAMS directory and got this:

```
$ python run_heat_exchanger.py 
2022-06-28 10:15:48 [INFO] idaes.init.fs.unit.shell: Initialization Complete
2022-06-28 10:15:48 [INFO] idaes.init.fs.unit.tube: Initialization Complete
2022-06-28 10:15:48 [INFO] idaes.init.fs.unit: Initialization Completed, optimal - Optimal Solution Found
Ipopt 3.13.2: nlp_scaling_method=gradient-based
tol=1e-06


******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit http://projects.coin-or.org/Ipopt

This version of Ipopt was compiled from source code available at
    https://github.com/IDAES/Ipopt as part of the Institute for the Design of
    Advanced Energy Systems Process Systems Engineering Framework (IDAES PSE
    Framework) Copyright (c) 2018-2019. See https://github.com/IDAES/idaes-pse.

This version of Ipopt was compiled using HSL, a collection of Fortran codes
    for large-scale scientific computation.  All technical papers, sales and
    publicity material resulting from use of the HSL codes within IPOPT must
    contain the following acknowledgement:
        HSL, a collection of Fortran codes for large-scale scientific
        computation. See http://www.hsl.rl.ac.uk.
******************************************************************************

This is Ipopt version 3.13.2, running with linear solver ma27.

Number of nonzeros in equality constraint Jacobian...:       21
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:       11

Total number of variables............................:       10
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        4
                     variables with only upper bounds:        0
Total number of equality constraints.................:       10
Total number of inequality constraints...............:        0
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  0.0000000e+00 9.21e-09 0.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0

Number of Iterations....: 0

                                   (scaled)                 (unscaled)
Objective...............:   0.0000000000000000e+00    0.0000000000000000e+00
Dual infeasibility......:   0.0000000000000000e+00    0.0000000000000000e+00
Constraint violation....:   1.8420210469596425e-11    9.2113623395562172e-09
Complementarity.........:   0.0000000000000000e+00    0.0000000000000000e+00
Overall NLP error.......:   1.8420210469596425e-11    9.2113623395562172e-09


Number of objective function evaluations             = 1
Number of objective gradient evaluations             = 1
Number of equality constraint evaluations            = 1
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 1
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 0
Total CPU secs in IPOPT (w/o function evaluations)   =      0.040
Total CPU secs in NLP function evaluations           =      0.000

EXIT: Optimal Solution Found.
Traceback (most recent call last):
  File "run_heat_exchanger.py", line 154, in <module>
    results = SolverFactory(solver_source).solve(
  File "/home/mmundt/devel-pyomo/pyomo-gams38/pyomo/solvers/plugins/solvers/GAMS.py", line 712, in solve
    self.available()
  File "/home/mmundt/devel-pyomo/pyomo-gams38/pyomo/solvers/plugins/solvers/GAMS.py", line 619, in available
    raise NameError(
NameError: 'gams' command failed to solve a simple model - GAMS shell solver functionality is not available.
```

Took me down the very wrong path of trying to figure out why the Python bindings and command path executable were not being found by Pyomo when in reality, the issue was that GAMS couldn't find the license.
Further update: The issue in #2446 was not related - this is due to the fact that the model had ExternalFunctions, which we do not support through the GAMS (and BARON) interface.

@renkekuhlmann - We have tried some models with GAMS 39.2.0 using `put_results='test_results'` (and both the `gdx` and `dat` formats), and we have not hit an error.
> We have tried some models with GAMS 39.2.0 using put_results='test_results' (and both the gdx and dat formats), and we have not hit an error.

I guess it very much depends on the numbers to write. It is expected that not all models will fail. The `gdx` path is not affected by this. Anyway, as described above, the fix is easy and doesn't hurt. Simply add an explicit space, e.g. 
```
output_file.write("\nput %s ' ' %s.l ' ' %s.m /;" % (var, var, var)) 
```
i.e., use a put statement like 
```
put x ' ' x.l ' ' x.m /;
```
instead of 
```
put x x.l x.m /;
```

Consider the following example:
```
Variable x;
x.l = 0.122323;
x.m = 1238721.3244;

file results /'puttest.txt'/;
results.nd=15;
results.nw=21;
put results;
put x x.l x.m /;
put x ' ' x.l ' ' x.m /;
```
The output is
```
x    0.1223230000000001.2387213244000000E+6
x     0.122323000000000 1.2387213244000000E+6
```
with GAMS 39.x.

Resolving patch:
diff --git a/pyomo/repn/plugins/gams_writer.py b/pyomo/repn/plugins/gams_writer.py
--- a/pyomo/repn/plugins/gams_writer.py
+++ b/pyomo/repn/plugins/gams_writer.py
@@ -813,9 +813,9 @@ def _write_model(self,
                 output_file.write("\nput results;")
                 output_file.write("\nput 'SYMBOL  :  LEVEL  :  MARGINAL' /;")
                 for var in var_list:
-                    output_file.write("\nput %s %s.l %s.m /;" % (var, var, var))
+                    output_file.write("\nput %s ' ' %s.l ' ' %s.m /;" % (var, var, var))
                 for con in constraint_names:
-                    output_file.write("\nput %s %s.l %s.m /;" % (con, con, con))
+                    output_file.write("\nput %s ' ' %s.l ' ' %s.m /;" % (con, con, con))
                 output_file.write("\nput GAMS_OBJECTIVE GAMS_OBJECTIVE.l "
                                   "GAMS_OBJECTIVE.m;\n")
 
@@ -826,7 +826,7 @@ def _write_model(self,
                 output_file.write("\nput statresults;")
                 output_file.write("\nput 'SYMBOL   :   VALUE' /;")
                 for stat in stat_vars:
-                    output_file.write("\nput '%s' %s /;\n" % (stat, stat))
+                    output_file.write("\nput '%s' ' ' %s /;\n" % (stat, stat))
 
 valid_solvers = {
 'ALPHAECP': {'MINLP','MIQCP'},


Test patch: