Repository: ietf-tools/datatracker

Issue:
referencedby link shows partial result only
### What happened?

https://datatracker.ietf.org/doc/draft-ietf-ace-aif/referencedby/

only shows draft-ietf-ace-key-groupcomm

It should be showing:

File: draft-ietf-ace-key-groupcomm
File: draft-ietf-ace-key-groupcomm-oscore
File: draft-ietf-ace-mqtt-tls-profile
File: draft-ietf-ace-oscore-gm-admin

I haven't checked with other documents.

### What browser(s) are you seeing the problem on?

Safari

### Code of Conduct

- [X] I agree to follow the IETF's Code of Conduct


Discussion:
A few more weird results:

https://datatracker.ietf.org/doc/rfc9031/references/

includes [draft-ietf-cbor-cddl](https://datatracker.ietf.org/doc/draft-ietf-cbor-cddl/) but doesn't include the RFC from that, RFC 8610 (which is what is actually being referenced from 9031).

In general, I'd like a referencedby query for RFC 8949 include those that reference RFC 7049 and vice versa, except when I don't (e.g., when I specifically look for documents that reference the earlier document).  And that includes the I-D stages of that document (e.g. draft-ietf-cbor-7049bis) and of the replaced document (7049, [draft-bormann-cbor](https://datatracker.ietf.org/doc/draft-bormann-cbor/)).  

But maybe that is a feature request, and we should get the direct references right, first.
This is a consequence of trying to take references out of xml when xml is provided, combined with the current state where we we are allowing xml to be submitted that isn't fully self-contained.

When attempting to rebuild the reference relationships for draft-ietf-ace-key-groupcomm-oscore using the xml in the archive for version 14:
```
{'warnings': ['There were 18 references with no matching DocAlias'], 'unfound': ['i-d.ietf-ace-oscore-profile', 'cose.algorithms', 'i-d.ietf-core-groupcomm-bis', 'nist-800-56a', 'i-d.ietf-ace-key-groupcomm', 'i-d.ietf-ace-oauth-authz', 'i-d.ietf-ace-oscore-gm-admin', 'i-d.ietf-cose-cbor-encoded-cert', 'cose.key.types', 'i-d.ietf-cose-rfc8152bis-algs', 'i-d.tiloca-core-oscore-discovery', 'cose.elliptic.curves', 'i-d.ietf-cose-rfc8152bis-struct', 'i-d.ietf-ace-dtls-authorize', 'cose.header.parameters', 'i-d.ietf-core-oscore-groupcomm', 'i-d.ietf-ace-aif', 'i-d.ietf-core-coap-pubsub']}
```

Whereas rebuilding it using the txt in the archive produces no warnings or errors.

We have a separate issue tracking when/how to require xml draft submissions to be self-contained.

In the meantime, we should change rebuild_reference_relations to look at the xml and see if it has xinclude elements, and if so, punt to the old mechanism of parsing the text. Then we need to build a management command to run through the documents that have been submitted in xml since we started building reference relations from the xml and reparse them.

Resolving patch:
diff --git a/ietf/doc/utils.py b/ietf/doc/utils.py
--- a/ietf/doc/utils.py
+++ b/ietf/doc/utils.py
@@ -763,9 +763,11 @@ def rebuild_reference_relations(doc, filenames):
     errors = []
     unfound = set()
     for ( ref, refType ) in refs.items():
-        # As of Dec 2021, DocAlias has a unique constraint on the name field, so count > 1 should not occur
-        refdoc = DocAlias.objects.filter( name=ref )
+        refdoc = DocAlias.objects.filter(name=ref)
+        if not refdoc and re.match(r"^draft-.*-\d{2}$", ref):
+            refdoc = DocAlias.objects.filter(name=ref[:-3])
         count = refdoc.count()
+        # As of Dec 2021, DocAlias has a unique constraint on the name field, so count > 1 should not occur
         if count == 0:
             unfound.add( "%s" % ref )
             continue
diff --git a/ietf/utils/xmldraft.py b/ietf/utils/xmldraft.py
--- a/ietf/utils/xmldraft.py
+++ b/ietf/utils/xmldraft.py
@@ -60,17 +60,47 @@ def cleanup():  # called when context exited, even if there's an exception
                 tree.tree = v2v3.convert2to3()
         return tree, xml_version
 
-    def _document_name(self, anchor):
-        """Guess document name from reference anchor
-
-        Looks for series numbers and removes leading 0s from the number.
-        """
-        anchor = anchor.lower()  # always give back lowercase
-        label = anchor.rstrip('0123456789')  # remove trailing digits
-        if label in ['rfc', 'bcp', 'fyi', 'std']:
-            number = int(anchor[len(label):])
-            return f'{label}{number}'
-        return anchor
+    def _document_name(self, ref):
+        """Get document name from reference."""
+        series = ["rfc", "bcp", "fyi", "std"]
+        # handle xinclude first
+        # FIXME: this assumes the xinclude is a bibxml href; if it isn't, there can
+        # still be false negatives. it would be better to expand the xinclude and parse
+        # its seriesInfo.
+        if ref.tag.endswith("}include"):
+            name = re.search(
+                rf"reference\.({'|'.join(series).upper()})\.(\d{{4}})\.xml",
+                ref.attrib["href"],
+            )
+            if name:
+                return f"{name.group(1)}{int(name.group(2))}".lower()
+            name = re.search(
+                r"reference\.I-D\.(?:draft-)?(.*)\.xml", ref.attrib["href"]
+            )
+            if name:
+                return f"draft-{name.group(1)}"
+            # can't extract the name, give up
+            return ""
+
+        # check the anchor next
+        anchor = ref.get("anchor").lower()  # always give back lowercase
+        label = anchor.rstrip("0123456789")  # remove trailing digits
+        if label in series:
+            number = int(anchor[len(label) :])
+            return f"{label}{number}"
+
+        # if we couldn't find a match so far, try the seriesInfo
+        series_query = " or ".join(f"@name='{x.upper()}'" for x in series)
+        for info in ref.xpath(
+            f"./seriesInfo[{series_query} or @name='Internet-Draft']"
+        ):
+            if not info.attrib["value"]:
+                continue
+            if info.attrib["name"] == "Internet-Draft":
+                return info.attrib["value"]
+            else:
+                return f'{info.attrib["name"].lower()}{info.attrib["value"]}'
+        return ""
 
     def _reference_section_type(self, section_name):
         """Determine reference type from name of references section"""
@@ -154,10 +184,20 @@ def get_refs(self):
         """Extract references from the draft"""
         refs = {}
         # accept nested <references> sections
-        for section in self.xmlroot.findall('back//references'):
-            ref_type = self._reference_section_type(self._reference_section_name(section))
-            for ref in (section.findall('./reference') + section.findall('./referencegroup')):
-                refs[self._document_name(ref.get('anchor'))] = ref_type
+        for section in self.xmlroot.findall("back//references"):
+            ref_type = self._reference_section_type(
+                self._reference_section_name(section)
+            )
+            for ref in (
+                section.findall("./reference")
+                + section.findall("./referencegroup")
+                + section.findall(
+                    "./xi:include", {"xi": "http://www.w3.org/2001/XInclude"}
+                )
+            ):
+                name = self._document_name(ref)
+                if name:
+                    refs[name] = ref_type
         return refs
 
 


Test patch:
diff --git a/ietf/utils/test_draft_with_references_v3.xml b/ietf/utils/test_draft_with_references_v3.xml
--- a/ietf/utils/test_draft_with_references_v3.xml
+++ b/ietf/utils/test_draft_with_references_v3.xml
@@ -1,6 +1,6 @@
 <?xml version='1.0'?>
 <?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
-<rfc category="exp" submissionType="independent" ipr="trust200902" docName="draft-test-references-00" version="3">
+<rfc xmlns:xi="http://www.w3.org/2001/XInclude" category="exp" submissionType="independent" ipr="trust200902" docName="draft-test-references-00" version="3">
     <front>
         <title>Test Draft with References</title>
         <author fullname="Alfred Person" initials="A." surname="Person" role="editor">
@@ -37,9 +37,25 @@
                 <seriesInfo name="RFC" value="1"/>
                 <seriesInfo name="DOI" value="10.17487/RFC0001"/>
             </reference>
+            <reference anchor="MNEMONIC" target="https://www.rfc-editor.org/info/rfc2">
+                <front>
+                    <title>Cloud Software</title>
+                    <author initials="D." surname="Crocker" fullname="D. Crocker">
+                        <organization/>
+                    </author>
+                    <date year="1969" month="April"/>
+                </front>
+                <seriesInfo name="RFC" value="2"/>
+                <seriesInfo name="DOI" value="10.17487/RFC0002"/>
+            </reference>
         </references>
         <references>
             <name>Informative References</name>
+            <xi:include href='https://xml.resource.org/public/rfc/bibxml3/reference.I-D.ietf-teas-pcecc-use-cases-00.xml'/>
+            <xi:include href='https://xml.resource.org/public/rfc/bibxml3/reference.I-D.ietf-teas-pcecc-use-cases.xml'/>
+            <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.draft-ietf-sipcore-multiple-reasons-00.xml" />
+            <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.draft-ietf-sipcore-multiple-reasons.xml" />
+            <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml9/reference.BCP.0014.xml" />
             <reference anchor='RFC0255' target='https://www.rfc-editor.org/info/rfc255'>
                 <front>
                     <title>Status of network hosts</title>
@@ -51,6 +67,34 @@
                 <seriesInfo name='RFC' value='255'/>
                 <seriesInfo name='DOI' value='10.17487/RFC0255'/>
             </reference>
+            <reference anchor="CONSISTENCY">
+              <front>
+                <title>Key Consistency and Discovery</title>
+                <author fullname="Alex Davidson" initials="A." surname="Davidson">
+                  <organization>Brave Software</organization>
+                </author>
+                <author fullname="Matthew Finkel" initials="M." surname="Finkel">
+                  <organization>The Tor Project</organization>
+                </author>
+                <author fullname="Martin Thomson" initials="M." surname="Thomson">
+                  <organization>Mozilla</organization>
+                </author>
+                <author fullname="Christopher A. Wood" initials="C. A." surname="Wood">
+                  <organization>Cloudflare</organization>
+                </author>
+                <date day="17" month="August" year="2022"/>
+                <abstract>
+                  <t>   This document describes the key consistency and correctness
+       requirements of protocols such as Privacy Pass, Oblivious DoH, and
+       Oblivious HTTP for user privacy.  It discusses several mechanisms and
+       proposals for enabling user privacy in varying threat models.  In
+       concludes with discussion of open problems in this area.
+
+                  </t>
+                </abstract>
+              </front>
+              <seriesInfo name="Internet-Draft" value="draft-wood-key-consistency-03"/>
+            </reference>
             <referencegroup anchor="bcp6">
                 <reference anchor='RFC1930' target='https://www.rfc-editor.org/info/rfc1930'>
                     <front>
@@ -191,4 +235,4 @@
             </reference>
         </references>
     </back>
-</rfc>
\ No newline at end of file
+</rfc>
diff --git a/ietf/utils/tests.py b/ietf/utils/tests.py
--- a/ietf/utils/tests.py
+++ b/ietf/utils/tests.py
@@ -374,10 +374,17 @@ def test_get_refs_v3(self):
             draft.get_refs(),
             {
                 'rfc1': XMLDraft.REF_TYPE_NORMATIVE,
+                'rfc2': XMLDraft.REF_TYPE_NORMATIVE,
+                'draft-wood-key-consistency-03': XMLDraft.REF_TYPE_INFORMATIVE,
                 'rfc255': XMLDraft.REF_TYPE_INFORMATIVE,
                 'bcp6': XMLDraft.REF_TYPE_INFORMATIVE,
+                'bcp14': XMLDraft.REF_TYPE_INFORMATIVE,
                 'rfc1207': XMLDraft.REF_TYPE_UNKNOWN,
                 'rfc4086': XMLDraft.REF_TYPE_NORMATIVE,
+                'draft-ietf-teas-pcecc-use-cases-00': XMLDraft.REF_TYPE_INFORMATIVE,
+                'draft-ietf-teas-pcecc-use-cases': XMLDraft.REF_TYPE_INFORMATIVE,
+                'draft-ietf-sipcore-multiple-reasons-00': XMLDraft.REF_TYPE_INFORMATIVE,
+                'draft-ietf-sipcore-multiple-reasons': XMLDraft.REF_TYPE_INFORMATIVE,
             }
         )