croqaz

/

experiment-llama-77

like 0

May be OCR or character encoding failures (e.g., UTF-8 misinterpreted as Latin-1) during the digitization of historical texts. Their presence in a tokenizer vocabulary is a critical data quality failure.
Abbé (1): Mojibake for "Abbé".
Franç (183): Mojibake for "Franç" (as in François or Française).

t-v4/tokenizer.json|jq . |rg 'Å|į|Ã|§|©'|rg :|cat -n

     1        "§": 116,
     2        "©": 118,
     3        "Ã": 141,
     4        "Å": 143,
     5        "į": 212,
     6        "é": 1185,
     7        "æ": 2338,
     8        "ĠÃ": 2803,
     9        "á": 3147,
    10        "è": 3858,
    11        "Åį": 3911,
    12        "ée": 4092,
    13        "â": 4106,
    14        "ĠÃł": 5346,
    15        "ê": 5429,
    16        "ÃŃ": 5642,
    17        "ä": 5726,
    18        "ô": 6193,
    19        "ó": 6900,
    20        "Ġ§": 7150,
    21        "Ãł": 7831,
    22        "Å«": 7860,
    23        "ö": 8111,
    24        "ü": 8132,
    25        "ĠcÅį": 8391,
    26        "ú": 9074,
    27        "ç": 9281,
    28        "Åĵ": 9546,
    29        "ñ": 10179,
    30        "Ġé": 10287,
    31        "Å·": 10340,
    32        "és": 10403,
    33        "ère": 10554,
    34        "§": 10785,
    35        "î": 11372,
    36        "ë": 12093,
    37        "ù": 12316,
    38        "ò": 12384,
    39        "ĠCæ": 12607,
    40        "ĠÃĨ": 12635,
    41        "û": 13092,
    42        "Ġdé": 13891,
    43        "Ġ§.": 14268,
    44        "ï": 14399,
    45        "ĠCæsar": 14694,
    46        "Ġsé": 14931,
    47        "ré": 14991,
    48        "án": 15415,
    49        "ÃĨ": 15464,
    50        "ð": 15762,
    51        "iÅį": 16016,
    52        "iæ": 16255,
    53        "Ġsée": 17517,
    54        "ét": 18150,
    55        "Ġâ": 18752,
    56        "én": 18808,
    57        "ár": 18854,
    58        "Å¿": 18900,
    59        "ées": 19118,
    60        "ès": 19420,
    61        "ĠfrÅį": 19566,
    62        "æus": 19624,
    63        "ôt": 19745,
    64        "Ġô": 19808,
    65        "Ġæ": 20445,
    66        "ité": 20446,
    67        "Ġré": 20621,
    68        "ête": 20727,
    69        "Ãī": 21025,
    70        "ér": 21843,
    71        "Ġþ": 22132,
    72        "Ġbée": 22964,
    73        "§.": 23292,
    74        "ês": 23330,
    75        "atiÅį": 23420,
    76        "Ġmé": 23482,
    77        "Ġhée": 23511,
    78        "Ġthée": 23764,
    79        "âte": 24165,
    80        "än": 24405,
    81        "áre": 24545,
    82        "ña": 24576,
    83        "éene": 24723,
    84        "ière": 24782,
    85        "Ġá": 24839,
    86        "idæ": 25067,
    87        "Ġné": 25463,
    88        "ĠcÅįm": 25480,
    89        "Ġét": 25670,
    90        "Ã¥": 25685,
    91        "Ġpræ": 25855,
    92        "ùm": 26041,
    93        "ân": 26134,
    94        "ĠAbbé": 26340,
    95        "æa": 26378,
    96        "él": 26456,
    97        "éd": 26475,
    98        "âd": 26659,
    99        "Ïį": 26679,
   100        "ême": 26729,
   101        "ĠFranç": 27358,
   102        "wÅ·": 27372,
   103        "Ãłm": 27591,
   104        "Ġfaç": 27916,
   105        "ères": 28028,
   106        "âteau": 28244,
   107        "ád": 28536,
   108        "ñor": 28642,
   109        "ÃŃa": 28759,
   110        "ĠÃĹ": 28976,
   111        "Ġè": 29455,
   112        "ür": 29542,
   113        "öl": 29904,
   114        "ég": 30113,
   115        "ón": 30215,
   116        "ĠMü": 30550,
   117        "Ġê": 30747,
   118        "Ġhæ": 30938,
   119        "ôn": 31075,
   120        "Ġthrée": 31166,
   121        "ŵ": 31235,
   122        "iæval": 31291,
   123        "Ġfé": 31306,
   124        "Ġwée": 31319,
   125        "Ġfaçade": 32055,
   126        "iâ": 32089,
   127        "æd": 32204,
   128        "ĠquÃłm": 32215,
   129        "ôtel": 32230,