Repository: ietf-tools/datatracker

Issue:
Many old draft versions missing from Datatracker
### Describe the issue

This issue appears to be related to #4898.

Several versions of older drafts do not appear in the Data Tracker UI. While they appear in the timeline diagram, they do not appear in the list of versions right above it, and clicking on the timeline to open the missing versions redirects to the latest publication instead.

This is a problem because some old drafts are still widely in use, such as SFTP version 3.

Examples:

- https://datatracker.ietf.org/doc/draft-ietf-secsh-filexfer/
- https://datatracker.ietf.org/doc/rfc3696/

Note that not all Internet Drafts are affected, such as https://datatracker.ietf.org/doc/rfc8989/

Workaround: Access the draft through https://www.ietf.org/archive/id/[name]-[version].txt

![image](https://user-images.githubusercontent.com/22013850/209896134-0d0a185e-6aea-43e8-bd08-0acb257ecb2a.png)

### Code of Conduct

- [X] I agree to follow the [IETF's Code of Conduct](https://github.com/ietf-tools/.github/blob/main/CODE_OF_CONDUCT.md)


Discussion:
This is a long standing issue rooted in there being a lack of information needed to create DocHistory objects for the revisions in question without making up information that the DocHistory should hold out of whole cloth. Specifically, the content of any of the metadata to be shown about the document reflecting the state it was in _at the time_ (such as who the shepherd was, if there was one, and whether there was WG state and what it was, etc.)

There are _many_ older documents affected by this, and any repair would require significant manual effort, and making up history that was never really true is almost unavoidable.



Can we detect these cases and add an alert to the page?
Yes, but it will cost time to do the computation - is it worth the extra lag?
(The computation is to check that there's a DocHistory record matching each NewRevisionDocEvent. There's a loop already that may be doing most of it where the `revisions` variable is assembled.)
This particular draft is annoying enough people (curious to know why now?) that an effort, preferably coordinated by the right AD,  to synthesize appropriate DocHistory records may be worth it. 
Per discussion at the tools team meeting, we will be continuing to investigate what we can do here without making up history. It may be that the fix will come with a general rework of the main document view at /doc/draft-.... to provide access to versions where the raw text exists but make it clear that there is no historical metadata available for that version.
I was unable to get back to https://github.com/ietf-tools/datatracker/issues/4936 in time, but i want to make sure its clear that the links in that issue and mentioned here (https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02) used to work not too long ago. We have a handful of docs that reference this link because it is the most widely supported SFTP spec and they are all broken. I created the bug issue when we first noticed, it is unclear how long ago this was changed, but it is not decades or even years, a more recent change broke this.
If its copy of draft-ietf-secsh-filexfer-02 has been deleted then perhaps it could at least stop returning draft-ietf-secsh-filexfer-13 instead, when asked for -02 (as described in #5518). The current behavior is actively misleading.

> This particular draft is annoying enough people (curious to know why now?) 

Why now: because it only just broke. It was working fine in October 2022.
- https://web.archive.org/web/20221007230001/https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02
- https://web.archive.org/web/20230104084838/https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02
Yes, and there are (as touched lightly on above) several courses of action we can take, _all_ of which have trade-offs.

The main doc page behavior is not different as far as I can reconstruct (or find in the wayback machine). If I'm right, here this became problematic is in the re-implementation of the /doc/html views. If we focused _there_ and allowed those views to point to versions of drafts we have, but don't have the metadata for, would the majority of the irritation the change is causing be alleviated?

I couldn't say what the biggest source of trouble for anyone else is. Still, I note that the [SFTP draft links in Wikipedia](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol#History_and_development) are currently broken too, so I think the #5518 variant of the issue is practical issue for more than just me.

(It's a shame that SFTP is specified only in expired drafts; but we're not going to solve that here.)
I mined the code deeply, and the main document page has _never_ returned anything but a redirect to version 13 for version 12 of this document. What changed is the behavior of the htmlized view at /doc/html/ for documents missing DocHistory. I think we should work towards a fix to _that_, cleaning the places in the new htmlization views that care about the metadata of the version being asked for.

Resolving patch:
diff --git a/ietf/doc/views_doc.py b/ietf/doc/views_doc.py
--- a/ietf/doc/views_doc.py
+++ b/ietf/doc/views_doc.py
@@ -244,7 +244,7 @@ def document_main(request, name, rev=None, document_html=False):
         is_author = request.user.is_authenticated and doc.documentauthor_set.filter(person__user=request.user).exists()
         can_view_possibly_replaces = can_edit_replaces or is_author
 
-        rfc_number = name[3:] if name.startswith("") else None
+        rfc_number = name[3:] if name.startswith("rfc") else None
         draft_name = None
         for a in aliases:
             if a.startswith("draft"):
@@ -487,13 +487,25 @@ def document_main(request, name, rev=None, document_html=False):
         html = None
         js = None
         css = None
+        diff_revisions = None
+        simple_diff_revisions = None
         if document_html:
-            html = doc.html_body()
-            if request.COOKIES.get("pagedeps") == "inline":
-                js = Path(finders.find("ietf/js/document_html.js")).read_text()
-                css = Path(finders.find("ietf/css/document_html_inline.css")).read_text()
-                if html:
-                    css += Path(finders.find("ietf/css/document_html_txt.css")).read_text()
+            diff_revisions=get_diff_revisions(request, name, doc if isinstance(doc,Document) else doc.doc)
+            simple_diff_revisions = [t[1] for t in diff_revisions]
+            simple_diff_revisions.reverse()
+            if not doc.is_rfc() and rev != doc.rev: 
+                # No DocHistory was found matching rev - snapshot will be false
+                # and doc will be a Document object, not a DocHistory
+                snapshot = True
+                doc = doc.fake_history_obj(rev)
+            else:
+                html = doc.html_body()
+                if request.COOKIES.get("pagedeps") == "inline":
+                    js = Path(finders.find("ietf/js/document_html.js")).read_text()
+                    css = Path(finders.find("ietf/css/document_html_inline.css")).read_text()
+                    if html:
+                        css += Path(finders.find("ietf/css/document_html_txt.css")).read_text()
+
         return render(request, "doc/document_draft.html" if document_html is False else "doc/document_html.html",
                                   dict(doc=doc,
                                        document_html=document_html,
@@ -505,7 +517,7 @@ def document_main(request, name, rev=None, document_html=False):
                                        name=name,
                                        content=content,
                                        split_content=split_content,
-                                       revisions=revisions,
+                                       revisions=simple_diff_revisions if document_html else revisions,
                                        snapshot=snapshot,
                                        stream_desc=stream_desc,
                                        latest_revision=latest_revision,
@@ -544,7 +556,7 @@ def document_main(request, name, rev=None, document_html=False):
                                        status_changes=status_changes,
                                        proposed_status_changes=proposed_status_changes,
                                        rfc_aliases=rfc_aliases,
-                                       has_errata=doc.tags.filter(slug="errata"),
+                                       has_errata=doc.pk and doc.tags.filter(slug="errata"), # doc.pk == None if using a fake_history_obj
                                        published=published,
                                        file_urls=file_urls,
                                        additional_urls=additional_urls,
@@ -569,7 +581,7 @@ def document_main(request, name, rev=None, document_html=False):
                                        review_assignments=review_assignments,
                                        no_review_from_teams=no_review_from_teams,
                                        due_date=due_date,
-                                       diff_revisions=get_diff_revisions(request, name, doc if isinstance(doc,Document) else doc.doc) if document_html else None
+                                       diff_revisions=diff_revisions
                                        ))
 
     if doc.type_id == "charter":
@@ -859,7 +871,7 @@ def document_html(request, name, rev=None):
     if not os.path.exists(doc.get_file_name()):
         raise Http404("File not found: %s" % doc.get_file_name())
 
-    return document_main(request, name=doc.name, rev=doc.rev if not doc.is_rfc() else None, document_html=True)
+    return document_main(request, name=doc.canonical_name(), rev=doc.rev if not doc.is_rfc() else None, document_html=True)
 
 def document_pdfized(request, name, rev=None, ext=None):
 


Test patch:
diff --git a/ietf/doc/tests.py b/ietf/doc/tests.py
--- a/ietf/doc/tests.py
+++ b/ietf/doc/tests.py
@@ -617,7 +617,9 @@ def setUp(self):
                 f.write(self.draft_text)
 
     def test_document_draft(self):
-        draft = WgDraftFactory(name='draft-ietf-mars-test',rev='01')
+        draft = WgDraftFactory(name='draft-ietf-mars-test',rev='01', create_revisions=range(0,2))
+
+
         HolderIprDisclosureFactory(docs=[draft])
         
         # Docs for testing relationships. Does not test 'possibly-replaces'. The 'replaced_by' direction
@@ -785,6 +787,7 @@ def test_document_draft(self):
         self.assertEqual(len(q('#sidebar option[value="draft-ietf-mars-test-00"][selected="selected"]')), 1)
 
         rfc = WgRfcFactory()
+        rfc.save_with_history([DocEventFactory(doc=rfc)])
         (Path(settings.RFC_PATH) / rfc.get_base_name()).touch()
         r = self.client.get(urlreverse("ietf.doc.views_doc.document_html", kwargs=dict(name=rfc.canonical_name())))
         self.assertEqual(r.status_code, 200)