Repository: SCons/scons

Issue:
lex and yacc emitters may get extra-files paths wrong
This is a summary of a Discord discussion (at https://discord.com/channels/571796279483564041/971831743944491008, although as a thread it probably won't survive and there's lots of other stuff in that thread - this bit should be listed as May 20, 2022).

The emitters in both the lex and yacc tools recognize certain flags, which will be passed on to the actual executed command, as requesting an additional file to be created, and grab that filename and add it to the target.  There's nothing wrong with that concept, but in doing so, they lose track of the fact that "shell paths" and "SCons paths" are not the same thing - the former are interpreted relative to the cwd of the executed command and the latter relative to the directory of the active SConscript.

The following simple subsidiary SConscript file illustrates this - `src/SConscript`:
```py
env = DefaultEnvironment()
env.Append(LEXFLAGS=['--header-file=gen-src/lexer.ll.hh'])
env.CXXFile('#gen-src/lexer.ll.cc', 'lexer.ll')
```
Calling the `CXXFile` builder sets up a target for the `.cc` file, and the emitter spots the `--header-file` option and adds another target for the rest of the option, `gen-src/lexer.ll.hh`.  However, the latter will be interpreted relative to the directory of the SConscript, so we get this tree:
```console
+-.
  +-SConstruct
  +-gen-src
  | +-gen-src/lexer.ll.cc
  |   +-src/lexer.ll
  |   +-/bin/flex
  +-src
    +-src/SConscript
    +-src/gen-src
    | +-src/gen-src/lexer.ll.hh
    |   +-src/lexer.ll
    |   +-/bin/flex
    +-src/lexer.ll

```
That is, the C++ source target is `gen-src/lexer.ll.cc`, while the C++ header target becomes `src/gen-src/lexer.ll.hh` because the SConscript is in `src`. For additional fun, the actual issued command is:
```bash
flex --header-file=gen-src/lexer.ll.hh -t src/lexer.ll > gen-src/lexer.ll.cc
```

meaning, we get SCons believing the header target is `src/gen-src/lexer.ll.hh` but the file was physically put in `gen-src/lexer.ll.hh` since the command-line flag is just passed on as it was set up.

Not sure whether the right answer is to adjust the two tools to do more figuring to get the paths consistent, or whether it's to leave the responsibility to the programmer to set the flags to be passed to lex/yacc in a way that avoids the problem. In the latter case, we probably need more documentation guidance. 

---

Here's some from the thread on generating the flags better:

> dmoody256: Try:
```py
env.Append(LEXFLAGS=['--header-file=${File("#gen-src/lexer.ll.hh")}'])
```

> NO_NAME: It does work for a predefined path but I have more lex files in my project. The actual line looks more like:
```py
env.Append(LEXFLAGS=['--header-file=gen-${SOURCE.path}.hh'])
```

> dmoody256: you can make a generator function then do all the work to resolve the string in a python function:
```py
def lex_header_gen(env, target, source, for_signature):
    result = f'--header-file=gen-{source[0]}.hh'
    return result

env['LEX_HEADER_GEN'] = lex_header_gen
env.Append(LEXFLAGS=['$LEX_HEADER_GEN'])
```
 
> NO_NAME: Wait, you can do that?!  Is this documented? How could I miss it?
> dmoody256: It is an extremely powerful feature
> NO_NAME: I genuinely cannot find it in the documentation.
> NO_NAME: I modified the code a little and it works:
```py
def lex_header_gen(env, target, source, for_signature):
    file = env.File(f'#gen-{source[0].path}.hh')
    result = f'--header-file={file}'
    return result

env['LEX_HEADER_GEN'] = lex_header_gen
env.Append(LEXFLAGS=['$LEX_HEADER_GEN'])
```

---

There don't seem to be any test cases that exercise the recognized "generate an extra file with a name chosen by the programmer" options, which are:
```py
fileGenOptions = ["--header-file=", "--tables-file="]   # lex tool
fileGenOptions = ["--defines=", "--graph="]  # yacc tool
```


Discussion:
There is a related problem on Stack Overflow: https://stackoverflow.com/questions/23007461/how-to-use-scons-lex-builder-with-variant-directories
The chosen solution should work with that case too.
Seems like just having the emitter create a File() node from the extracted string (parameter to the argument), perhaps needs to be subst()'d first. Would allow specifying both relative `path/to/file` to current SConscript (the default for an unadorned path), and top of tree relative using `#path/to/file`

So:
```py
            l = len(fileGenOption)
            if option[:l] == fileGenOption:
                # A file generating option is present, so add the
                # file name to the target list.
                fileName = option[l:].strip()
                target.append(fileName)
```
Does it make a difference if that's `target.append(File(fileName))`? Seems like SCons will just do that conversion anyway?
The difference is that you could then do `#path/to/file` and it should work which won't work with just a string.
Though I suppose the command string would be broken.
HRm.
Let me ponder and respond later.
Related, but not the same problem: #4168 - related because it also has to do with the use of a string extracted from the `*FLAGS` variable and then somehow turning that into something SCons tracks properly.
Will add a more concise explanation later, but this simple stanza seems to work out:
```py
env.Append(LEXFLAGS=['--header-file=${TARGET.base}.hh'])
```

Here's my attempt at a concise analysis (written for lex, but applies to some extent to yacc/bison as well):
* The default `LEXCOM` on Linux is: `"$LEX $LEXFLAGS -t $SOURCES > $TARGET"`
* If `LEXFLAGS` contains file-output lex options, the emitter set up by the `lex.py` tool detects them, and adds any option-argument to the target list.
* The targets (including any added by the emitter) are interpreted by SCons relative to the sconscript directory.
* The actual issued commands are executed relative to the top directory, with the sources and targets adjusted by SCons.
* `lex` just does what it's told in the arguments, so unless the contents of `LEXFLAGS` have been modified, we get a mismatch.

This is an even more pruned-down example - only a subdir is needed to trigger the mismatch. variantdir is a red herring, although since a variantdir always involves a subdir to build in, it does always trigger it.
```py
$ cat SConstruct
SConscript('src/SConscript')
$ cat src/SConscript
env = Environment()
env.Append(LEXFLAGS=['--header-file=mylex.l.h'])
env.CFile('mylex.l.c', 'mylex.l')
$ scons --tree=prune
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
flex --header-file=mylex.l.h -t src/mylex.l > src/mylex.l.c     # source/target adjusted, header-file arg is not
+-.
  +-SConstruct
  +-src
    +-src/SConscript
    +-src/mylex.l
    +-src/mylex.l.c
    | +-src/mylex.l
    | +-/bin/flex
    +-src/mylex.l.h       # the scons tracked header is relative to the SConscript dir
      +-src/mylex.l
      +-/bin/flex
scons: done building targets.
$ ls -R
mylex.l.h  SConstruct src    # lex writes hdr into topdir, where it ran from

./src:
mylex.l  mylex.l.c  SConscript   # the lex-generated C file is the right place, SCons adjusted path
```
So in a discord discussion (adding here, so it's captured) we mused about defining construction variables for the two lex file-generation options and the two main bison file-generation options (which don't exist in POSIX yacc, but that's another topic).  Oddly, it picks out these:
```py
    # With --defines and --graph, the name of the file is totally defined
    # in the options.
    fileGenOptions = ["--defines=", "--graph="]
```
Bison documents `-g, --graph`, but no longer documents `--defines` in the manpage. In the info page we see:
```txt
--defines[=file]
    Historical name for option --header before Bison 3.8.
```
3.8 is actually relatively new... Sept 2021, so maybe the yacc tool should look for both names?
> So in a discord discussion (adding here, so it's captured) we mused about defining construction variables for the two lex file-generation options and the two main bison file-generation options (which don't exist in POSIX yacc, but that's another topic). Oddly, it picks out these:
> 
> ```python
>     # With --defines and --graph, the name of the file is totally defined
>     # in the options.
>     fileGenOptions = ["--defines=", "--graph="]
> ```
> 
> Bison documents `-g, --graph`, but no longer documents `--defines` in the manpage. In the info page we see:
> 
> ```
> --defines[=file]
>     Historical name for option --header before Bison 3.8.
> ```
> 
> 3.8 is actually relatively new... Sept 2021, so maybe the yacc tool should look for both names?

Yup. The joys of change.. ;)
So to try and narrow in on a preferred solution: if new construction variables are added, just leave current behavior of sniffing in `LEXFLAGS` and `YACCFLAGS` alone, but update the doc to say the other method is preferred (and possibly, why)? Update (pressed button too early) does there need to be any detection of duplication between flag and consvar?
Right. The idea is to not break existing builds (and require a deprecation cycle)..
This is similar to users adding `-Ipath1` to CCFLAGS instead of via CPPPATH.
There may be reasons to do that.

No need to add duplication checking.

Resolving patch:
diff --git a/SCons/Tool/lex.py b/SCons/Tool/lex.py
--- a/SCons/Tool/lex.py
+++ b/SCons/Tool/lex.py
@@ -80,6 +80,18 @@ def lexEmitter(target, source, env) -> tuple:
                 file_name = option[l:].strip()
                 target.append(file_name)
 
+    lexheaderfile = env.subst("$LEX_HEADER_FILE", target=target, source=source)
+    if lexheaderfile:
+        target.append(lexheaderfile)
+        # rewrite user-supplied file string with a node, we need later
+        env.Replace(LEX_HEADER_FILE=env.File(lexheaderfile))
+
+    lextablesfile = env.subst("$LEX_TABLES_FILE", target=target, source=source)
+    if lextablesfile:
+        target.append(lextablesfile)
+        # rewrite user-supplied file string with a node, we need later
+        env.Replace(LEX_TABLES_FILE=env.File(lextablesfile))
+
     return target, source
 
 
@@ -129,18 +141,25 @@ def generate(env) -> None:
     cxx_file.add_action(".ll", LexAction)
     cxx_file.add_emitter(".ll", lexEmitter)
 
-    env["LEXFLAGS"] = CLVar("")
-
     if sys.platform == 'win32':
-        # ignore the return - we do not need the full path here
+        # ignore the return, all we need is for the path to be added
         _ = get_lex_path(env, append_paths=True)
-        env["LEX"] = env.Detect(BINS)
-        if not env.get("LEXUNISTD"):
-            env["LEXUNISTD"] = CLVar("")
-        env["LEXCOM"] = "$LEX $LEXUNISTD $LEXFLAGS -t $SOURCES > $TARGET"
+
+    env.SetDefault(
+        LEX=env.Detect(BINS),
+        LEXFLAGS=CLVar(""),
+        LEX_HEADER_FILE="",
+        LEX_TABLES_FILE="",
+    )
+
+    if sys.platform == 'win32':
+        env.SetDefault(LEXUNISTD=CLVar(""))
+        env["LEXCOM"] = "$LEX $LEXUNISTD $LEXFLAGS $_LEX_HEADER $_LEX_TABLES -t $SOURCES > $TARGET"
     else:
-        env["LEX"] = env.Detect(BINS)
-        env["LEXCOM"] = "$LEX $LEXFLAGS -t $SOURCES > $TARGET"
+        env["LEXCOM"] = "$LEX $LEXFLAGS $_LEX_HEADER $_LEX_TABLES -t $SOURCES > $TARGET"
+
+    env['_LEX_HEADER'] = '${LEX_HEADER_FILE and "--header-file=" + str(LEX_HEADER_FILE)}'
+    env['_LEX_TABLES'] = '${LEX_TABLES_FILE and "--tables-file=" + str(LEX_TABLES_FILE)}'
 
 
 def exists(env) -> Optional[str]:
diff --git a/SCons/Tool/yacc.py b/SCons/Tool/yacc.py
--- a/SCons/Tool/yacc.py
+++ b/SCons/Tool/yacc.py
@@ -98,6 +98,18 @@ def _yaccEmitter(target, source, env, ysuf, hsuf) -> tuple:
                 fileName = option[l:].strip()
                 target.append(fileName)
 
+    yaccheaderfile = env.subst("$YACC_HEADER_FILE", target=target, source=source)
+    if yaccheaderfile:
+        target.append(yaccheaderfile)
+        # rewrite user-supplied file string with a node, we need later
+        env.Replace(YACC_HEADER_FILE=env.File(yaccheaderfile))
+
+    yaccgraphfile = env.subst("$YACC_GRAPH_FILE", target=target, source=source)
+    if yaccgraphfile:
+        target.append(yaccgraphfile)
+        # rewrite user-supplied file string with a node, we need later
+        env.Replace(YACC_GRAPH_FILE=env.File(yaccgraphfile))
+
     return target, source
 
 
@@ -163,14 +175,19 @@ def generate(env) -> None:
         # ignore the return, all we need is for the path to be added
         _ = get_yacc_path(env, append_paths=True)
 
-    if 'YACC' not in env:
-        env["YACC"] = env.Detect(BINS)
+    env.SetDefault(
+        YACC=env.Detect(BINS),
+        YACCFLAGS=CLVar(""),
+        YACC_HEADER_FILE="",
+        YACC_GRAPH_FILE="",
+    )
 
-    env['YACCFLAGS'] = CLVar('')
-    env['YACCCOM'] = '$YACC $YACCFLAGS -o $TARGET $SOURCES'
+    env['YACCCOM'] = '$YACC $YACCFLAGS $_YACC_HEADER $_YACC_GRAPH -o $TARGET $SOURCES'
     env['YACCHFILESUFFIX'] = '.h'
     env['YACCHXXFILESUFFIX'] = '.hpp'
     env['YACCVCGFILESUFFIX'] = '.vcg'
+    env['_YACC_HEADER'] = '${YACC_HEADER_FILE and "--header=" + str(YACC_HEADER_FILE)}'
+    env['_YACC_GRAPH'] = '${YACC_GRAPH_FILE and "--graph=" + str(YACC_GRAPH_FILE)}'
 
 
 def exists(env) -> Optional[str]:


Test patch:
diff --git a/test/LEX/FLEXFLAGS.py b/test/LEX/FLEXFLAGS.py
--- a/test/LEX/FLEXFLAGS.py
+++ b/test/LEX/FLEXFLAGS.py
@@ -25,9 +25,11 @@
 
 """
 Test that detection of file-writing options in LEXFLAGS works.
+Also test that construction vars for the same purpose work.
 """
 
 import sysconfig
+from pathlib import Path
 
 import TestSCons
 from TestCmd import IS_WINDOWS
@@ -37,16 +39,18 @@
 
 test = TestSCons.TestSCons()
 
-test.subdir('sub')
+test.subdir('sub1')
+test.subdir('sub2')
 
 test.file_fixture('mylex.py')
 
 test.write('SConstruct', """\
 DefaultEnvironment(tools=[])
-SConscript("sub/SConscript")
+SConscript(dirs=['sub1', 'sub2'])
 """)
 
-test.write(['sub', 'SConscript'], f"""\
+# this SConscript is for the options-in-flags version
+test.write(['sub1', 'SConscript'], f"""\
 import sys
 
 env = Environment(
@@ -60,7 +64,25 @@
 if not all((len(t) == 3, "header.h" in t, "tables.t" in t)):
     sys.exit(1)
 """)
-test.write(['sub', 'aaa.l'], "aaa.l\nLEXFLAGS\n")
+test.write(['sub1', 'aaa.l'], "aaa.l\nLEXFLAGS\n")
+
+# this SConscript is for the construction var version
+test.write(['sub2', 'SConscript'], f"""\
+import sys
+
+env = Environment(
+    LEX=r'{_python_} mylex.py',
+    LEXFLAGS='-x',
+    tools=['default', 'lex'],
+)
+env.CFile(
+    target='aaa',
+    source='aaa.l',
+    LEX_HEADER_FILE='header.h',
+    LEX_TABLES_FILE='tables.t',
+)
+""")
+test.write(['sub2', 'aaa.l'], "aaa.l\nLEXFLAGS\n")
 
 test.run('.', stderr=None)
 
@@ -69,19 +91,38 @@
     lexflags = ' --nounistd' + lexflags
 # Read in with mode='r' because mylex.py implicitly wrote to stdout
 # with mode='w'.
-test.must_match(['sub', 'aaa.c'], "aaa.l\n%s\n" % lexflags, mode='r')
+test.must_match(['sub1', 'aaa.c'], "aaa.l\n%s\n" % lexflags, mode='r')
 
 # NOTE: this behavior is "wrong" but we're keeping it for compat.
-# the generated files should go into 'sub'.
+# the generated files should go into 'sub1', not the topdir.
 test.must_match(['header.h'], 'lex header\n')
 test.must_match(['tables.t'], 'lex table\n')
 
 # To confirm the files from the file-output options were tracked,
-# do a clean and make sure they got removed. As noted, they currently
-# don't go into the tracked location, so using the the SConscript check instead.
+# we should do a clean and make sure they got removed.
+# As noted, they currently don't go into the tracked location,
+# so using the check in the SConscript instead.
 #test.run(arguments='-c .')
-#test.must_not_exist(test.workpath(['sub', 'header.h']))
-#test.must_not_exist(test.workpath(['sub', 'tables.t']))
+#test.must_not_exist(test.workpath(['sub1', 'header.h']))
+#test.must_not_exist(test.workpath(['sub1', 'tables.t']))
+
+sub2 = Path('sub2')
+headerfile = sub2 / 'header.h'
+tablefile = sub2 / 'tables.t'
+lexflags = f' -x --header-file={headerfile} --tables-file={tablefile} -t'
+if IS_WINDOWS and not sysconfig.get_platform() in ("mingw",):
+    lexflags = ' --nounistd' + lexflags
+# Read in with mode='r' because mylex.py implicitly wrote to stdout
+# with mode='w'.
+test.must_match(['sub2', 'aaa.c'], "aaa.l\n%s\n" % lexflags, mode='r')
+test.must_match(['sub2', 'header.h'], 'lex header\n')
+test.must_match(['sub2', 'tables.t'], 'lex table\n')
+
+# To confirm the files from the file-output options were tracked,
+# do a clean and make sure they got removed.
+test.run(arguments='-c .', stderr=None)
+test.must_not_exist(test.workpath('sub2', 'header.h'))
+test.must_not_exist(test.workpath('sub2', 'tables.t'))
 
 test.pass_test()
 
diff --git a/test/YACC/BISONFLAGS.py b/test/YACC/BISONFLAGS.py
--- a/test/YACC/BISONFLAGS.py
+++ b/test/YACC/BISONFLAGS.py
@@ -25,8 +25,11 @@
 
 """
 Test that detection of file-writing options in YACCFLAGS works.
+Also test that the construction vars for the same purpose work.
 """
 
+from pathlib import Path
+
 import TestSCons
 from TestCmd import IS_WINDOWS
 
@@ -42,16 +45,18 @@
 
 test = TestSCons.TestSCons()
 
-test.subdir('sub')
+test.subdir('sub1')
+test.subdir('sub2')
 
 test.dir_fixture('YACCFLAGS-fixture')
 
 test.write('SConstruct', """\
 DefaultEnvironment(tools=[])
-SConscript("sub/SConscript")
+SConscript(dirs=['sub1', 'sub2'])
 """)
 
-test.write(['sub', 'SConscript'], """\
+# this SConscript is for the options-in-flags version
+test.write(['sub1', 'SConscript'], """\
 import sys
 
 env = Environment(
@@ -65,21 +70,56 @@
 if not all((len(t) == 3, "header.h" in t, "graph.g" in t)):
     sys.exit(1)
 """ % locals())
-test.write(['sub', 'aaa.y'], "aaa.y\nYACCFLAGS\n")
+test.write(['sub1', 'aaa.y'], "aaa.y\nYACCFLAGS\n")
+
+# this SConscript is for the construction var version
+test.write(['sub2', 'SConscript'], """\
+import sys
+
+env = Environment(
+    YACC=r'%(_python_)s myyacc.py',
+    YACCFLAGS='-x',
+    tools=['yacc', '%(linker)s', '%(compiler)s'],
+)
+env.CFile(
+    target='aaa',
+    source='aaa.y',
+    YACC_HEADER_FILE='header.h',
+    YACC_GRAPH_FILE='graph.g',
+)
+""" % locals())
+test.write(['sub2', 'aaa.y'], "aaa.y\nYACCFLAGS\n")
+
 test.run('.', stderr=None)
-test.must_match(['sub', 'aaa.c'], "aaa.y\n -x --header=header.h --graph=graph.g\n")
+test.must_match(['sub1', 'aaa.c'], "aaa.y\n -x --header=header.h --graph=graph.g\n")
 
 # NOTE: this behavior is "wrong" but we're keeping it for compat:
-# the generated files should go into 'sub'.
+# the generated files should go into 'sub1', not the topdir.
 test.must_match(['header.h'], 'yacc header\n')
 test.must_match(['graph.g'], 'yacc graph\n')
 
+# To confirm the files from the file-output options were tracked,
+# we should do a clean and make sure they got removed.
+# As noted, they currently don't go into the tracked location,
+# so using the check in the SConscript instead.
+#test.run(arguments='-c .')
+#test.must_not_exist(test.workpath(['sub1', 'header.h']))
+#test.must_not_exist(test.workpath(['sub1', 'graph.g']))
+
+sub2 = Path('sub2')
+headerfile = sub2 / 'header.h'
+graphfile = sub2 / 'graph.g'
+yaccflags = f"aaa.y\n -x --header={headerfile} --graph={graphfile}\n"
+test.must_match(['sub2', 'aaa.c'], yaccflags)
+test.must_match(['sub2', 'header.h'], 'yacc header\n')
+test.must_match(['sub2', 'graph.g'], 'yacc graph\n')
+
 # To confirm the files from the file-output options were tracked,
 # do a clean and make sure they got removed. As noted, they currently
 # don't go into the tracked location, so using the the SConscript check instead.
-#test.run(arguments='-c .')
-#test.must_not_exist(test.workpath(['sub', 'header.h']))
-#test.must_not_exist(test.workpath(['sub', 'graph.g']))
+test.run(arguments='-c .')
+test.must_not_exist(test.workpath('sub2', 'header.h'))
+test.must_not_exist(test.workpath('sub2', 'graph.g'))
 
 test.pass_test()
 
diff --git a/test/YACC/YACCFLAGS-fixture/myyacc.py b/test/YACC/YACCFLAGS-fixture/myyacc.py
--- a/test/YACC/YACCFLAGS-fixture/myyacc.py
+++ b/test/YACC/YACCFLAGS-fixture/myyacc.py
@@ -6,7 +6,7 @@
 def make_side_effect(path, text):
     p = Path(path)
     if str(p.parent) != '.':
-        p.mkdir(parents=True, exist_ok=True)
+        p.parent.mkdir(parents=True, exist_ok=True)
     with p.open(mode="wb") as f:
         f.write(text)
 
diff --git a/test/fixture/mylex.py b/test/fixture/mylex.py
--- a/test/fixture/mylex.py
+++ b/test/fixture/mylex.py
@@ -15,7 +15,7 @@
 def make_side_effect(path, text):
     p = Path(path)
     if str(p.parent) != '.':
-        p.mkdir(parents=True, exist_ok=True)
+        p.parent.mkdir(parents=True, exist_ok=True)
     with p.open(mode="wb") as f:
         f.write(text)