Repository: ietf-tools/datatracker

Issue:
htmlization page for rfc441 is 500ing
Unicode decode error that does not occur in the dev machines given the same input.


Discussion:
Note that the RFC contains:
```
          [This RFC was put into machine readable form for entry]
      [into the online RFC archives by Hélène Morin, Viagénie, 12/99]
```
Not sure that's what's irritating the bug, but it's a good place to look.
Similar at /doc/pdf/rfc1305
@rjsparks is this with `feat/htmlize` or the old code?
Ah, the new one.

There are quite a few RFCs that seem to have UTF-8 issues:
``` shell
# find . -type f -iname '*txt' -exec isutf8 {} +
./rfc177.txt: line 485, char 56, byte 20365: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc2497.txt: line 210, char 69, byte 7748: Expecting bytes in the following ranges: 00..7F C2..F4.
./rfc2708.txt: line 521, char 15, byte 21995: After a first byte between C2 and DF, expecting a 2nd byte between 80 and BF
./rfc367.txt: line 184, char 39, byte 7911: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc2875.txt: line 754, char 13, byte 25274: After a first byte of F4, expecting 2nd byte between 80 and 8F.
./rfc316.txt: line 369, char 40, byte 15366: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc288.txt: line 211, char 57, byte 8400: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc101.txt: line 753, char 58, byte 30224: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc2557.txt: line 813, char 35, byte 34544: After a first byte between C2 and DF, expecting a 2nd byte between 80 and BF
./rfc303.txt: line 607, char 57, byte 18096: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc317.txt: line 51, char 40, byte 1483: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc315.txt: line 181, char 41, byte 7669: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc310.txt: line 389, char 40, byte 18485: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc270.txt: line 46, char 58, byte 1273: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc313.txt: line 440, char 40, byte 19767: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc307.txt: line 331, char 39, byte 11695: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc1305.txt: line 2551, char 16, byte 121366: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc306.txt: line 189, char 40, byte 7345: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc323.txt: line 499, char 39, byte 15666: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc64.txt: line 65, char 57, byte 2491: Expecting bytes in the following ranges: 00..7F C2..F4.
./rfc243.txt: line 353, char 57, byte 12303: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc2166.txt: line 641, char 20, byte 29258: Expecting bytes in the following ranges: 00..7F C2..F4.
./rfc282.txt: line 446, char 57, byte 18010: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc292.txt: line 529, char 58, byte 22223: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc290.txt: line 828, char 57, byte 34229: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc327.txt: line 277, char 40, byte 11684: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc441.txt: line 234, char 18, byte 9173: Expecting bytes in the following ranges: 00..7F C2..F4.
./rfc234.txt: line 53, char 58, byte 1543: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc235.txt: line 243, char 57, byte 7869: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc369.txt: line 602, char 40, byte 24212: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc182.txt: line 52, char 57, byte 2471: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc237.txt: line 50, char 40, byte 2101: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc178.txt: line 607, char 59, byte 24417: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
./rfc227.txt: line 70, char 57, byte 2321: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
```

Resolving patch:
diff --git a/ietf/doc/models.py b/ietf/doc/models.py
--- a/ietf/doc/models.py
+++ b/ietf/doc/models.py
@@ -550,7 +550,7 @@ def html_body(self, classes=""):
                 html = Path(
                     os.path.join(settings.RFC_PATH, self.canonical_name() + ".html")
                 ).read_text()
-            except IOError:
+            except (IOError, UnicodeDecodeError):
                 return None
         else:
             try:
@@ -560,7 +560,7 @@ def html_body(self, classes=""):
                         self.name + "-" + self.rev + ".html",
                     )
                 ).read_text()
-            except IOError:
+            except (IOError, UnicodeDecodeError):
                 return None
 
         # If HTML was generated by rfc2html, do not return it. Caller


Test patch: