Repository: SCons/scons

Issue:
doc generation with lxml fails on construction variable links
When building the generated files which will be used for the manpage and User Guide builds:
```
python3 bin/docs-update-generated.py
```
the result has problems if lxml is in use.  Since SCons has shifted to Python 3 support exclusively, and the bindings for libxslt (which can be used together with the libxml2 bindings instead of lxml) are officially unsupported on Python 3, the use of lxml should be considered the "normal" case.

Specifically, when the Tool documents are generated (`tools.gen` and `tools.mod`), references to SCons construction variables are extracted from the xml files and turned into xml entities that represent links, and will eventually show up as live links in the html output documents.  These references take the form of "Sets" and "Uses" references, like this snip from `gcc.xml`:
```
<sets>
<item>CC</item>
<item>SHCCFLAGS</item>
<item>CCVERSION</item>
</sets>
```
The actual links are generated in `bin/scons-proc.py`, in the `write_gen` method of the `SConsXML` class, like this:
```
            if len(v.sets):
                added = True
                vp = stf.newNode("para")
                s = ['&cv-link-%s;' % x for x in v.sets]
                stf.setText(vp, 'Sets:  ' + ', '.join(s) + '.')
                stf.appendNode(vl, vp)
            if len(v.uses):
                added = True
                vp = stf.newNode("para")
                u = ['&cv-link-%s;' % x for x in v.uses]
                stf.setText(vp, 'Uses:  ' + ', '.join(u) + '.')
                stf.appendNode(vl, vp)
```
So the gcc entry should come out like this:
```
<para>Sets:  &cv-link-CC;, &cv-link-CCVERSION;, &cv-link-SHCCFLAGS;.</para>
```
But when lxml is in use, the `&` is coverted to a character entity so we get this, which is broken:
```
<para>Sets:  &amp;cv-link-CC;, &amp;cv-link-CCVERSION;, &amp;cv-link-SHCCFLAGS;.</para>
```
The code above is apparently the only place we write strings to nodes that contain what look like entities (`&foo;`) so it seems to be the only place that suffers from this XML escaping problem.

At the moment a simple post-processing step (e.g. run `sed` to turn `&amp;cv-` back into `&cv-`) can be used (no such workaround is integrated into the build, has been done manually), but we need to figure out how to fix this "more correctly". There is an entirely different definition of the `TreeFactory` class in `bin/SConsDoc.py` for the lxml case vs. the libxml2 case, presumably this is where something could happen, but I can't see what, yet.


Discussion:
As a cross-reference, it seems there's another doc problem with Py3/lxml: #3584 
Problem is reproducible under Ubuntu 19.10 and Fedora 30.
The basic culprit seems to be that the `tostring` in `SConsDoc::writeGenTree` escapes the ampersands, see https://stackoverflow.com/questions/27340099/printing-html-entities-using-lxml-in-python , for example.
Should we be using: http://www.sagehill.net/docbookxsl/Db5Tools.html#Db5UnivLinking instead of the &XYZ inside <para>?
@mwichmann 's PR is a hack to work around this issue.
It works and let's us move on, but I'm going to reopen as an enhancement request.
I'm thinking if we used the docbook link per URL above and created the right node structure this would work. That the issue if we used the docbook link inside a text is xml should encode it.

Probably the way to figure this out is to parse the example 

```
<para>Use the
<command xlink:href="#ref-preview">Preview</command>
command to generate a preview.</para>
```

with lxml and see what the node structure looks like, we'd know how to add such programmatically in our existing code.

Resolving patch:
diff --git a/bin/docs-update-generated.py b/bin/docs-update-generated.py
--- a/bin/docs-update-generated.py
+++ b/bin/docs-update-generated.py
@@ -51,13 +51,9 @@ def generate_all():
                              '-v', argpair('variables')] + flist,
                             shell=False)
 
-        cp.check_returncode()  # bail if it failed
-        # lxml: fixup possibly broken tools.gen:
-        with open(os.path.join(gen_folder, 'tools.gen'), 'r') as f :
-            filedata = f.read()
-        filedata = filedata.replace(r'&amp;cv-link', r'&cv-link')
-        with open(os.path.join(gen_folder, 'tools.gen'), 'w') as f :
-            f.write(filedata)
+        # No-op: scons-proc doesn't actually set an exit code at the moment.
+        if cp.returncode:
+            print("Generation failed", file=sys.stderr)
     
     
 if __name__ == "__main__":
diff --git a/bin/scons-proc.py b/bin/scons-proc.py
--- a/bin/scons-proc.py
+++ b/bin/scons-proc.py
@@ -141,23 +141,31 @@ def write_gen(self, filename):
                     added = True
                     stf.appendNode(vl, stf.copyNode(s))
             
+            # Generate the text for sets/uses lists of construction vars.
+            # This used to include an entity reference which would be replaced
+            # by the link to the cvar, but with lxml, dumping out the tree
+            # with tostring() will encode the & introducing the entity,
+            # breaking it. Instead generate the actual link. (issue #3580)
             if v.sets:
                 added = True
                 vp = stf.newNode("para")
-                # if using lxml, the &entity; entries will be encoded,
-                # effectively breaking them.  should fix,
-                # for now handled post-process in calling script.
-                s = ['&cv-link-%s;' % x for x in v.sets]
-                stf.setText(vp, 'Sets:  ' + ', '.join(s) + '.')
+                stf.setText(vp, "Sets: ")
+                for setv in v.sets:
+                    link = stf.newSubNode(vp, "link", linkend="cv-%s" % setv)
+                    linktgt = stf.newSubNode(link, "varname")
+                    stf.setText(linktgt, "$" + setv)
+                    stf.setTail(link, " ")
                 stf.appendNode(vl, vp)
+
             if v.uses:
                 added = True
                 vp = stf.newNode("para")
-                # if using lxml, the &entity; entries will be encoded,
-                # effectively breaking them.  should fix,
-                # for now handled post-process in calling script.
-                u = ['&cv-link-%s;' % x for x in v.uses]
-                stf.setText(vp, 'Uses:  ' + ', '.join(u) + '.')
+                stf.setText(vp, "Uses: ")
+                for use in v.uses:
+                    link = stf.newSubNode(vp, "link", linkend="cv-%s" % use)
+                    linktgt = stf.newSubNode(link, "varname")
+                    stf.setText(linktgt, "$" + use)
+                    stf.setTail(link, " ")
                 stf.appendNode(vl, vp)
                 
             # Still nothing added to this list item?


Test patch: