Repository: SCons/scons

Issue:
doc generation with lxml fails on construction variable links
When building the generated files which will be used for the manpage and User Guide builds:
```
python3 bin/docs-update-generated.py
```
the result has problems if lxml is in use.  Since SCons has shifted to Python 3 support exclusively, and the bindings for libxslt (which can be used together with the libxml2 bindings instead of lxml) are officially unsupported on Python 3, the use of lxml should be considered the "normal" case.

Specifically, when the Tool documents are generated (`tools.gen` and `tools.mod`), references to SCons construction variables are extracted from the xml files and turned into xml entities that represent links, and will eventually show up as live links in the html output documents.  These references take the form of "Sets" and "Uses" references, like this snip from `gcc.xml`:
```
<sets>
<item>CC</item>
<item>SHCCFLAGS</item>
<item>CCVERSION</item>
</sets>
```
The actual links are generated in `bin/scons-proc.py`, in the `write_gen` method of the `SConsXML` class, like this:
```
            if len(v.sets):
                added = True
                vp = stf.newNode("para")
                s = ['&cv-link-%s;' % x for x in v.sets]
                stf.setText(vp, 'Sets:  ' + ', '.join(s) + '.')
                stf.appendNode(vl, vp)
            if len(v.uses):
                added = True
                vp = stf.newNode("para")
                u = ['&cv-link-%s;' % x for x in v.uses]
                stf.setText(vp, 'Uses:  ' + ', '.join(u) + '.')
                stf.appendNode(vl, vp)
```
So the gcc entry should come out like this:
```
<para>Sets:  &cv-link-CC;, &cv-link-CCVERSION;, &cv-link-SHCCFLAGS;.</para>
```
But when lxml is in use, the `&` is coverted to a character entity so we get this, which is broken:
```
<para>Sets:  &amp;cv-link-CC;, &amp;cv-link-CCVERSION;, &amp;cv-link-SHCCFLAGS;.</para>
```
The code above is apparently the only place we write strings to nodes that contain what look like entities (`&foo;`) so it seems to be the only place that suffers from this XML escaping problem.

At the moment a simple post-processing step (e.g. run `sed` to turn `&amp;cv-` back into `&cv-`) can be used (no such workaround is integrated into the build, has been done manually), but we need to figure out how to fix this "more correctly". There is an entirely different definition of the `TreeFactory` class in `bin/SConsDoc.py` for the lxml case vs. the libxml2 case, presumably this is where something could happen, but I can't see what, yet.


Discussion:
As a cross-reference, it seems there's another doc problem with Py3/lxml: #3584 
Problem is reproducible under Ubuntu 19.10 and Fedora 30.
The basic culprit seems to be that the `tostring` in `SConsDoc::writeGenTree` escapes the ampersands, see https://stackoverflow.com/questions/27340099/printing-html-entities-using-lxml-in-python , for example.
Should we be using: http://www.sagehill.net/docbookxsl/Db5Tools.html#Db5UnivLinking instead of the &XYZ inside <para>?

Resolving patch:
diff --git a/bin/docs-update-generated.py b/bin/docs-update-generated.py
--- a/bin/docs-update-generated.py
+++ b/bin/docs-update-generated.py
@@ -43,13 +43,21 @@ def generate_all():
                 print("Couldn't create destination folder %s! Exiting..." % gen_folder)
                 return
         # Call scons-proc.py
-        _ = subprocess.call([sys.executable,
-                              os.path.join('bin','scons-proc.py'),
-                              '-b', argpair('builders'),
-                              '-f', argpair('functions'),
-                              '-t', argpair('tools'),
-                              '-v', argpair('variables')] + flist,
-                             shell=False)
+        cp = subprocess.run([sys.executable,
+                             os.path.join('bin','scons-proc.py'),
+                             '-b', argpair('builders'),
+                             '-f', argpair('functions'),
+                             '-t', argpair('tools'),
+                             '-v', argpair('variables')] + flist,
+                            shell=False)
+
+        cp.check_returncode()  # bail if it failed
+        # lxml: fixup possibly broken tools.gen:
+        with open(os.path.join(gen_folder, 'tools.gen'), 'r') as f :
+            filedata = f.read()
+        filedata = filedata.replace(r'&amp;cv-link', r'&cv-link')
+        with open(os.path.join(gen_folder, 'tools.gen'), 'w') as f :
+            f.write(filedata)
     
     
 if __name__ == "__main__":
diff --git a/bin/scons-proc.py b/bin/scons-proc.py
--- a/bin/scons-proc.py
+++ b/bin/scons-proc.py
@@ -144,12 +144,18 @@ def write_gen(self, filename):
             if v.sets:
                 added = True
                 vp = stf.newNode("para")
+                # if using lxml, the &entity; entries will be encoded,
+                # effectively breaking them.  should fix,
+                # for now handled post-process in calling script.
                 s = ['&cv-link-%s;' % x for x in v.sets]
                 stf.setText(vp, 'Sets:  ' + ', '.join(s) + '.')
                 stf.appendNode(vl, vp)
             if v.uses:
                 added = True
                 vp = stf.newNode("para")
+                # if using lxml, the &entity; entries will be encoded,
+                # effectively breaking them.  should fix,
+                # for now handled post-process in calling script.
                 u = ['&cv-link-%s;' % x for x in v.uses]
                 stf.setText(vp, 'Uses:  ' + ', '.join(u) + '.')
                 stf.appendNode(vl, vp)


Test patch: