Repository: Pyomo/pyomo

Issue:
create_instance is failing on master
To replicate:
cd pyomo/examples/pyomo/tutorial
python set.py

The final error is from the dataportal, but this error first cropped up after the last release. Anyway, here is the last line of the stack trace:
File "/home/woodruff/software/DLWFORKS/pyomo/pyomo/dataportal/parse_datacmds.py", line 452, in p_error
    raise IOError(tmp)
OSError: Syntax error at token 'NUM_VAL' with value '1' (line 1, column 9)

This affects a lot of legacy models.



Discussion:
Does this still happen if you clean out *.pyc files? Because this should be failing some tests (even ones that don't require solvers like the pyomo.gdp.tests.test_gdp.Reformulate ones), but it actually isn't. I was seeing test failures from it locally, but after some messing around I can't reproduce it anymore, even with same Python and ply versions.
I'm able to recreate this error with Pyomo master on Linux even after clearing *.pyc files.
Also, when I wrote "cropped up after the last release" I meant "cropped up since the last release" (The release is OK)
I also get stuck with a `model.create_instance` error.

To replicate
1. First install pysperf(https://github.com/grossmann-group/pysperf)
2. Put the following in `~/bin/pysperf`
```
#!python path
from pysperf.pysperf_main import parse_command_line_arguments_and_run
parse_command_line_arguments_and_run()
```
3. run `pysperf list models` in the terminal.

The error is

```
Traceback (most recent call last):
  File "/Users/zedongpeng/bin/pysperf", line 2, in <module>
    from pysperf.pysperf_main import parse_command_line_arguments_and_run
  File "/Users/zedongpeng/Github/pysperf/pysperf/__init__.py", line 1, in <module>
    from . import model_library
  File "/Users/zedongpeng/Github/pysperf/pysperf/model_library.py", line 11, in <module>
    compute_model_stats()
  File "/Users/zedongpeng/Github/pysperf/pysperf/model_library_tools.py", line 90, in compute_model_stats
    pyomo_model = test_model.build_function()
  File "/Users/zedongpeng/Github/pysperf/pysperf/model_library_tools.py", line 55, in build_function_with_BM_suffix
    pyomo_model = build_function()
  File "/Users/zedongpeng/Github/pyomo/examples/gdp/stickies.py", line 492, in build_model
    instance = model.create_instance(os.path.join(this_file_dir(), DATFILE))
  File "/Users/zedongpeng/Github/pyomo/pyomo/core/base/PyomoModel.py", line 726, in create_instance
    profile_memory=profile_memory )
  File "/Users/zedongpeng/Github/pyomo/pyomo/core/base/PyomoModel.py", line 760, in load
    dp = DataPortal(filename=arg, model=self)
  File "/Users/zedongpeng/Github/pyomo/pyomo/dataportal/DataPortal.py", line 67, in __init__
    self.load()
  File "/Users/zedongpeng/Github/pyomo/pyomo/dataportal/DataPortal.py", line 164, in load
    status = self._data_manager.process(self._model, self._data, self._default)
  File "/Users/zedongpeng/Github/pyomo/pyomo/dataportal/plugins/datacommands.py", line 62, in process
    _process_include(['include', self.filename], model, data, default, self.options)
  File "/Users/zedongpeng/Github/pyomo/pyomo/dataportal/process_data.py", line 616, in _process_include
    scenarios = parse_data_commands(filename=cmd[1])
  File "/Users/zedongpeng/Github/pyomo/pyomo/dataportal/parse_datacmds.py", line 530, in parse_data_commands
    dat_yaccer.parse(data, lexer=dat_lexer, debug=debug)
  File "/Users/zedongpeng/opt/anaconda3/lib/python3.7/site-packages/ply/yacc.py", line 333, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/Users/zedongpeng/opt/anaconda3/lib/python3.7/site-packages/ply/yacc.py", line 1201, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/Users/zedongpeng/opt/anaconda3/lib/python3.7/site-packages/ply/yacc.py", line 192, in call_errorfunc
    r = errorfunc(token)
  File "/Users/zedongpeng/Github/pyomo/pyomo/dataportal/parse_datacmds.py", line 452, in p_error
    raise IOError(tmp)
OSError: Syntax error at token 'NUM_VAL' with value '0.1' (line 8, column 13)
```

@ZedongPeng Haha, thanks for putting this here. No need to install `pysperf` to replicate. All it is doing is importing and attempting to call the model construction function in `examples.gdp.stickies`.
In case it helps, I did a little bit of digging using `examples.gdp.jobshop`, and it seems to hang on the `1` of the `STAGES` definition. My ply version `ply-3.11` is the same as on the Github actions.

Data file:
```
set JOBS := A B C ;
set STAGES := 1 2 3 ;

param tau:
      1  2  3  :=
  A   5  .  3 
  B   .  3  2 
  C   2  4  . ;
```

Ply debug output:
```
PLY: PARSE DEBUG START

State  : 0
Stack  : . LexToken(SET,'set',2,1)
Action : Shift and goto state 5

State  : 5
Stack  : SET . LexToken(WORD,'JOBS',2,5)
Action : Shift and goto state 16

State  : 16
Stack  : SET WORD . LexToken(COLONEQ,':=',2,10)
Action : Shift and goto state 47

State  : 47
Stack  : SET WORD COLONEQ . LexToken(WORD,'A',2,13)
Action : Shift and goto state 87

State  : 87
Stack  : SET WORD COLONEQ WORD . LexToken(WORD,'B',2,15)
Action : Reduce rule [data -> WORD] with ['A'] and goto state 89
Result : <list @ 0x7f22f1747ac8> (['A'])

State  : 89
Stack  : SET WORD COLONEQ data . LexToken(WORD,'B',2,15)
Action : Shift and goto state 110

State  : 110
Stack  : SET WORD COLONEQ data WORD . LexToken(WORD,'C',2,17)
Action : Reduce rule [data -> data WORD] with [['A'],'B'] and goto state 89
Result : <list @ 0x7f22f1747ac8> (['A', 'B'])

State  : 89
Stack  : SET WORD COLONEQ data . LexToken(WORD,'C',2,17)
Action : Shift and goto state 110

State  : 110
Stack  : SET WORD COLONEQ data WORD . LexToken(SEMICOLON,';',2,19)
Action : Reduce rule [data -> data WORD] with [['A', 'B'],'C'] and goto state 89
Result : <list @ 0x7f22f1747ac8> (['A', 'B', 'C'])

State  : 89
Stack  : SET WORD COLONEQ data . LexToken(SEMICOLON,';',2,19)
Action : Reduce rule [datastar -> data] with [['A', 'B', 'C']] and goto state 88
Result : <list @ 0x7f22f1747ac8> (['A', 'B', 'C'])

State  : 88
Stack  : SET WORD COLONEQ datastar . LexToken(SEMICOLON,';',2,19)
Action : Shift and goto state 109

State  : 109
Stack  : SET WORD COLONEQ datastar SEMICOLON . LexToken(SET,'set',3,21)
Action : Reduce rule [statement -> SET WORD COLONEQ datastar SEMICOLON] with ['set','JOBS',':=',['A', 'B', 'C'],';'] and goto state 3
Result : <list @ 0x7f22f1747a48> (['set', 'JOBS', ':=', 'A', 'B', 'C'])

State  : 3
Stack  : statement . LexToken(SET,'set',3,21)
Action : Reduce rule [statements -> statement] with [<list @ 0x7f22f1747a48>] and goto state 2
Result : <list @ 0x7f22f1747b08> ([['set', 'JOBS', ':=', 'A', 'B', 'C']])

State  : 2
Stack  : statements . LexToken(SET,'set',3,21)
Action : Shift and goto state 5

State  : 5
Stack  : statements SET . LexToken(WORD,'STAGES',3,25)
Action : Shift and goto state 16

State  : 16
Stack  : statements SET WORD . LexToken(COLONEQ,':=',3,32)
Action : Shift and goto state 47

State  : 47
Stack  : statements SET WORD COLONEQ . LexToken(NUM_VAL,1,3,35)
ERROR: Error  : statements SET WORD COLONEQ . LexToken(NUM_VAL,1,3,35)
```
Ok, a bit more digging, because I got curious. It looks like line 1072 of `yacc.py` in `ply` that we end up with a `IndexError` on numeric types:
```
# Check the action table
ltype = lookahead.type
t = actions[state].get(ltype)
```
Here, `actions[state]` is a dict containing keys `INT_VAL` and `FLOAT_VAL`, but `ltype` is `NUM_VAL`.
This causes `t` to be `None`, and we get the error.
Sorry @jsiirola, it looks like it works on 12f1cbde96f1b020b328a1dc2fb6ca55c2de09a1 and fails on a0633b974e5cc4ede82ef9253b15d6d97048bd01.
Can you please locate *and delete* the files `parse_table_datacmds.py` and `parse_table_datacmds.pyc` and see if that clears things up?
@jsiirola that fixed it for me
@DLWoodruff @qtothec @ZedongPeng did John's suggestion fix this issue for you?
Yes. John's suggestion fixes the issue for me.
I'm glad this fixes things.  That said, it is curious as to why we need to manually delete those files.  In theory, PLY is supposed to delete/refresh those files automatically when the parse rules change.
It also works for me. Thank you, John.
@blnicho Yes, it fixed it for me as well. This would make sense why it doesn't crop up on new environments. Would this also be a problem for people who did `pip install --upgrade pyomo`?
@qtothec yeah it might be a problem for anyone using dat files that doesn't get Pyomo 5.6.9 from a fresh install. Given that I expect this is a very small subset of people I'm going to close this since we've figured out a solution to the original issue. 

Resolving patch:
diff --git a/pyomo/dataportal/parse_datacmds.py b/pyomo/dataportal/parse_datacmds.py
--- a/pyomo/dataportal/parse_datacmds.py
+++ b/pyomo/dataportal/parse_datacmds.py
@@ -12,6 +12,7 @@
 
 import bisect
 import sys
+import logging
 import os
 import os.path
 import ply.lex as lex
@@ -22,7 +23,7 @@
 from pyutilib.misc import flatten_list, import_file
 
 from pyomo.common import config
-from pyomo.common.fileutils import this_file_dir
+from pyomo.common.fileutils import this_file_dir, this_file
 
 _re_number = r'[-+]?(?:[0-9]+\.?[0-9]*|\.[0-9]+)(?:[eE][-+]?[0-9]+)?'
 
@@ -461,6 +462,7 @@ def p_error(p):
 
 dat_lexer = None
 dat_yaccer = None
+dat_yaccer_tabfile = None
 
 #
 # The function that performs the parsing
@@ -469,13 +471,36 @@ def parse_data_commands(data=None, filename=None, debug=0, outputdir=None):
 
     global dat_lexer
     global dat_yaccer
+    global dat_yaccer_tabfile
 
     if outputdir is None:
         # Try and write this into the module source...
         outputdir = os.path.dirname(getfile( currentframe() ))
+        _tabfile = os.path.join(outputdir, tabmodule+".py")
         # Ideally, we would pollute a per-user configuration directory
         # first -- something like ~/.pyomo.
         if not os.access(outputdir, os.W_OK):
+            _file = this_file()
+            logger = logging.getLogger('pyomo.dataportal')
+
+            if os.path.exists(_tabfile) and \
+               os.path.getmtime(_file) >= os.path.getmtime(_tabfile):
+                logger.warning(
+                    "Potentially outdated DAT Parse Table found in source "
+                    "tree (%s), but you do not have write access to that "
+                    "directory, so we cannot update it.  Please notify "
+                    "you system administrator to remove that file"
+                    % (_tabfile,))
+            if os.path.exists(_tabfile+'c') and \
+               os.path.getmtime(_file) >= os.path.getmtime(_tabfile+'c'):
+                logger.warning(
+                    "Potentially outdated DAT Parse Table found in source "
+                    "tree (%s), but you do not have write access to that "
+                    "directory, so we cannot update it.  Please notify "
+                    "you system administrator to remove that file"
+                    % (_tabfile+'c',))
+
+            # Switch the directory for the tabmodule to the current directory
             outputdir = os.getcwd()
 
     # if the lexer/yaccer haven't been initialized, do so.
@@ -484,17 +509,26 @@ def parse_data_commands(data=None, filename=None, debug=0, outputdir=None):
         # Always remove the parser.out file, which is generated to
         # create debugging
         #
-        if os.path.exists("parser.out"):        #pragma:nocover
-            os.remove("parser.out")
-        if debug > 0:                           #pragma:nocover
+        _parser_out = os.path.join(outputdir, "parser.out")
+        if os.path.exists(_parser_out):
+            os.remove(_parser_out)
+
+        _tabfile = dat_yaccer_tabfile = os.path.join(outputdir, tabmodule+".py")
+        if debug > 0 or \
+           ( os.path.exists(_tabfile) and
+             os.path.getmtime(__file__) >= os.path.getmtime(_tabfile) ):
             #
             # Remove the parsetab.py* files.  These apparently need to
             # be removed to ensure the creation of a parser.out file.
             #
-            if os.path.exists(tabmodule+".py"):
-                os.remove(tabmodule+".py")
-            if os.path.exists(tabmodule+".pyc"):
-                os.remove(tabmodule+".pyc")
+            if os.path.exists(_tabfile):
+                os.remove(_tabfile)
+            if os.path.exists(_tabfile+"c"):
+                os.remove(_tabfile+"c")
+
+            for _mod in list(sys.modules.keys()):
+                if _mod == tabmodule or _mod.endswith('.'+tabmodule):
+                    del sys.modules[_mod]
 
         dat_lexer = lex.lex()
         #


Test patch:
diff --git a/pyomo/dataportal/tests/test_dat_parser.py b/pyomo/dataportal/tests/test_dat_parser.py
new file mode 100644
--- /dev/null
+++ b/pyomo/dataportal/tests/test_dat_parser.py
@@ -0,0 +1,34 @@
+#  ___________________________________________________________________________
+#
+#  Pyomo: Python Optimization Modeling Objects
+#  Copyright 2017 National Technology and Engineering Solutions of Sandia, LLC
+#  Under the terms of Contract DE-NA0003525 with National Technology and 
+#  Engineering Solutions of Sandia, LLC, the U.S. Government retains certain 
+#  rights in this software.
+#  This software is distributed under the 3-clause BSD License.
+#  ___________________________________________________________________________
+#
+
+import os
+import time
+
+import pyutilib.th as unittest
+
+import pyomo.dataportal.parse_datacmds as parser
+
+class TestDatParser(unittest.TestCase):
+    def test_update_parsetable(self):
+        parser.parse_data_commands('')
+        self.assertIsNotNone(parser.dat_yaccer)
+        _tabfile = parser.dat_yaccer_tabfile
+        mtime = os.path.getmtime(_tabfile)
+        if _tabfile[-1] == 'c':
+            _tabfile = _tabfile[:-1]
+        time.sleep(0.01)
+        with open(parser.__file__, 'a'):
+            os.utime(parser.__file__, None)
+        parser.dat_lexer = None
+        parser.dat_yaccer = None
+        parser.parse_data_commands('')
+        self.assertIsNotNone(parser.dat_yaccer)
+        self.assertLess(mtime, os.path.getmtime(_tabfile))