Repository: ietf-tools/datatracker

Issue:
importing minutes for session incorrectly compares new and old documents
`type_defect`   |    by jennifer@painless-security.com

___


The `import_session_minutes()` view compares a copy of the document available from notes.ietf.org against the existing minutes for the session. If these are unchanged, a warning is displayed and the "Import" button to confirm the import is disabled.

The handling of `de-gfm` is incorrect in this comparison. The data pulled from notes.ietf.org are run through `de-gfm` (by the `Note` class) and compared against the existing version after running it through `de-gfm` as well. This is wrong because the existing version was already run through `de-gfm` before it was saved in the first place.

I haven't poked at `de-gfm` enough to know exactly why, but for at least some cases running it twice produces changed output.

The upshot of this is that a user will not be prevented from importing a redundant revision of the minutes. The data they are shown as a preview and that are saved on import are consistent and correct.

___
_Issue migrated from trac:3558 at 2022-03-04 09:29:18 +0000_


Discussion:
_@jennifer@painless-security.com_ _uploaded file [`no-double-degfm.patch`](https://raw.githubusercontent.com/ietf-tools/svntrac-converted-attachments/main/datatracker/attachments/ticket/314/3141dfc5f8ca656e9c87a3fd89961e509e3d124c/da9fd2fa47a2874121f3d8beb60a39a2e0559aa7.patch) (0.8 KiB)_

Fix for the issue
_@jennifer@painless-security.com_ _commented_


___
(I'll deal with the patch but I'm in the middle of some other work - posting it here so it doesn't get lost)
[attachments-ticket-3558.zip](https://github.com/ietf-tools/datatracker/files/8188758/attachments-ticket-3558.zip)


Resolving patch:
diff --git a/ietf/meeting/views.py b/ietf/meeting/views.py
--- a/ietf/meeting/views.py
+++ b/ietf/meeting/views.py
@@ -4169,11 +4169,11 @@ def import_session_minutes(request, session_id, num):
     if current_minutes:
         try:
             with open(current_minutes.get_file_name()) as f:
-                if import_contents == Note.preprocess_source(f.read()):
+                if import_contents == f.read():
                     contents_changed = False
                     messages.warning(request, 'This document is identical to the current revision, no need to import.')
-        except FileNotFoundError:
-            pass  # allow import if the file is missing
+        except Exception:
+            pass  # Do not let a failure here prevent minutes from being updated.
 
     return render(
         request,


Test patch:
diff --git a/ietf/meeting/tests_views.py b/ietf/meeting/tests_views.py
--- a/ietf/meeting/tests_views.py
+++ b/ietf/meeting/tests_views.py
@@ -6109,17 +6109,47 @@ def test_refuses_identical_import(self):
                          kwargs={'num': self.meeting.number, 'session_id': self.session.pk})
 
         self.client.login(username='secretary', password='secretary+password')
-        r = self.client.post(url, {'markdown_text': 'original markdown text'})  # create a rev
-        self.assertEqual(r.status_code, 302)
         with requests_mock.Mocker() as mock:
             mock.get(f'https://notes.ietf.org/{self.session.notes_id()}/download', text='original markdown text')
             mock.get(f'https://notes.ietf.org/{self.session.notes_id()}/info',
                      text=json.dumps({"title": "title", "updatetime": "2021-12-02T11:22:33z"}))
+            # Create a revision. Run the original text through the preprocessing done when importing
+            # from the notes site.
+            r = self.client.get(url)  # let GET do its preprocessing
+            q = PyQuery(r.content)
+            r = self.client.post(url, {'markdown_text': q('input[name="markdown_text"]').attr['value']})
+            self.assertEqual(r.status_code, 302)
+
             r = self.client.get(url)  # try to import the same text
             self.assertContains(r, "This document is identical", status_code=200)
             q = PyQuery(r.content)
             self.assertEqual(len(q('button:disabled[type="submit"]')), 1)
-            self.assertEqual(len(q('button:not(:disabled)[type="submit"]')), 0)
+            self.assertEqual(len(q('button:enabled[type="submit"]')), 0)
+
+    def test_allows_import_on_existing_bad_unicode(self):
+        """Should not be able to import text identical to the current revision"""
+        url = urlreverse('ietf.meeting.views.import_session_minutes',
+                         kwargs={'num': self.meeting.number, 'session_id': self.session.pk})
+
+        self.client.login(username='secretary', password='secretary+password')
+        r = self.client.post(url, {'markdown_text': 'replaced below'})  # create a rev
+        with open(
+                self.session.sessionpresentation_set.filter(document__type="minutes").first().document.get_file_name(),
+                'wb'
+        ) as f:
+            # Replace existing content with an invalid Unicode byte string. The particular invalid
+            # values here are accented characters in the MacRoman charset (see ticket #3756).
+            f.write(b'invalid \x8e unicode \x99\n')
+        self.assertEqual(r.status_code, 302)
+        with requests_mock.Mocker() as mock:
+            mock.get(f'https://notes.ietf.org/{self.session.notes_id()}/download', text='original markdown text')
+            mock.get(f'https://notes.ietf.org/{self.session.notes_id()}/info',
+                     text=json.dumps({"title": "title", "updatetime": "2021-12-02T11:22:33z"}))
+            r = self.client.get(url)  # try to import the same text
+            self.assertNotContains(r, "This document is identical", status_code=200)
+            q = PyQuery(r.content)
+            self.assertEqual(len(q('button:enabled[type="submit"]')), 1)
+            self.assertEqual(len(q('button:disabled[type="submit"]')), 0)
 
     def test_handles_missing_previous_revision_file(self):
         """Should still allow import if the file for the previous revision is missing"""