Package: tokenizer
OverviewTokenization is a necessary first step in many natural language processing
tasks, such as word counting, parsing, spell checking, corpus generation, and
statistical analysis of text.Tokenizeris a compact pure-Python (>= 3.8) executable
program and module for tokenizing Icelandic text. It converts input text to
streams oftokens, where each token is a separate word, punctuation sign,
number/amount, date, e-mail, URL/URI, etc. It also segments the token stream
into sentences, considering corner cases such as abbreviations and dates in
the middle of sentences.The package contains a dictionary of common Icelandic abbreviations,
in the filesrc/tokenizer/Abbrev.conf.Tokenizer is an independent spinoff from theGreynir project(GitHub repositoryhere), by the same authors.
TheGreynir natural language parser for Icelandicuses Tokenizer on its input.Note that Tokenizer is licensed under theMITlicense
while Greynir is licensed underGPLv3.Deep vs. shallow tokenizationTokenizer can do bothdeepandshallowtokenization.Shallowtokenization simply returns each sentence as a string (or as a line
of text in an output file), where the individual tokens are separated
by spaces.Deeptokenization returns token objects that have been annotated with
the token type and further information extracted from the token, for example
a(year, month, day)tuple in the case of date tokens.In shallow tokenization, tokens are in most cases kept intact, although
consecutive white space is always coalesced. The input strings"800 MW","21. janúar"and"800 7000"thus become
two tokens each, output with a single space between them.In deep tokenization, the same strings are represented by single token objects,
of typeTOK.MEASUREMENT,TOK.DATERELandTOK.TELNO, respectively.
The text associated with a single token object may contain spaces,
although consecutive whitespace is always coalesced into a single space" ".By default, the command line tool performs shallow tokenization. If you
want deep tokenization with the command line tool, use the--jsonor--csvswitches.From Python code, callsplit_into_sentences()for shallow tokenization,
ortokenize()for deep tokenization. These functions are documented with
examples below.InstallationTo install:$pipinstalltokenizerCommand line toolAfter installation, the tokenizer can be invoked directly from
the command line:$tokenizeinput.txtoutput.txtInput and output files are in UTF-8 encoding. If the files are not
given explicitly,stdinandstdoutare used for input and output,
respectively.Empty lines in the input are treated as hard sentence boundaries.By default, the output consists of one sentence per line, where each
line ends with a single newline character (ASCII LF,chr(10),"\n").
Within each line, tokens are separated by spaces.The following (mutually exclusive) options can be specified
on the command line:--csvDeep tokenization. Output token objects in CSV
format, one per line. Sentences are separated by
lines containing0,"",""--jsonDeep tokenization. Output token objects in JSON
format, one per line.Other options can be specified on the command line:-n--normalizeNormalize punctuation, causing e.g. quotes to be
output in Icelandic form and hyphens to be
regularized. This option is only applicable to
shallow tokenization.-s--one_sent_per_lineInput contains strictly one sentence per line,
i.e. every newline is a sentence boundary.-o--originalOutput original token text, i.e. bypass shallow
tokenization. This effectively runs the tokenizer
as a sentence splitter only.-m--convert_measurementsDegree signal in tokens denoting temperature
normalized (200° C -> 200 °C)-p--coalesce_percentNumbers combined into one token with the
following token denoting percentage word forms
(prósent,prósentustig,hundraðshlutar)-g--keep_composite_glyphsDo not replace composite glyphs using Unicode
COMBINING codes with their accented/umlaut
counterparts-e--replace_html_escapesHTML escape codes replaced by their meaning,
such as&aacute;->á-c--convert_numbersEnglish-style decimal points and thousands
separators in numbers changed to Icelandic style-kN--handle_kludgy_ordinalsNKludgy ordinal handling defined.
0: Returns the original mixed word form
1. Kludgy ordinal returned as pure word forms
2: Kludgy ordinals returned as pure numbersTypetokenize-hortokenize--helpto get a short help message.Example$echo"3.janúar sl. keypti   ég 64kWst rafbíl. Hann kostaði € 30.000."|tokenize3. janúar sl. keypti ég 64kWst rafbíl .
Hann kostaði €30.000 .$echo"3.janúar sl. keypti   ég 64kWst rafbíl. Hann kostaði € 30.000."|tokenize--csv19,"3. janúar","0|1|3"
6,"sl.","síðastliðinn"
6,"keypti",""
6,"ég",""
22,"64kWst","J|230400000.0"
6,"rafbíl",""
1,".","."
0,"",""
6,"Hann",""
6,"kostaði",""
13,"€30.000","30000|EUR"
1,".","."
0,"",""$echo"3.janúar sl. keypti   ég 64kWst rafbíl. Hann kostaði € 30.000."|tokenize--json{"k":"BEGIN SENT"}
{"k":"DATEREL","t":"3. janúar","v":[0,1,3]}
{"k":"WORD","t":"sl.","v":["síðastliðinn"]}
{"k":"WORD","t":"keypti"}
{"k":"WORD","t":"ég"}
{"k":"MEASUREMENT","t":"64kWst","v":["J",230400000.0]}
{"k":"WORD","t":"rafbíl"}
{"k":"PUNCTUATION","t":".","v":"."}
{"k":"END SENT"}
{"k":"BEGIN SENT"}
{"k":"WORD","t":"Hann"}
{"k":"WORD","t":"kostaði"}
{"k":"AMOUNT","t":"€30.000","v":[30000,"EUR"]}
{"k":"PUNCTUATION","t":".","v":"."}
{"k":"END SENT"}Python moduleShallow tokenization exampleAn example of shallow tokenization from Python code goes something like this:fromtokenizerimportsplit_into_sentences# A string to be tokenized, containing two sentencess="3.janúar sl. keypti   ég 64kWst rafbíl. Hann kostaði € 30.000."# Obtain a generator of sentence stringsg=split_into_sentences(s)# Loop through the sentencesforsentenceing:# Obtain the individual token stringstokens=sentence.split()# Print the tokens, comma-separatedprint("|".join(tokens))The program outputs:3.|janúar|sl.|keypti|ég|64kWst|rafbíl|.
Hann|kostaði|€30.000|.Deep tokenization exampleTo do deep tokenization from within Python code:fromtokenizerimporttokenize,TOKtext=("Málinu var vísað til stjórnskipunar- og eftirlitsnefndar ""skv. 3. gr. XVII. kafla laga nr. 10/2007 þann 3. janúar 2010.")fortokenintokenize(text):print("{0}: '{1}'{2}".format(TOK.descr[token.kind],token.txtor"-",token.valor""))Output:BEGIN SENT: '-' (0, None)
WORD: 'Málinu'
WORD: 'var'
WORD: 'vísað'
WORD: 'til'
WORD: 'stjórnskipunar- og eftirlitsnefndar'
WORD: 'skv.' [('samkvæmt', 0, 'fs', 'skst', 'skv.', '-')]
ORDINAL: '3.' 3
WORD: 'gr.' [('grein', 0, 'kvk', 'skst', 'gr.', '-')]
ORDINAL: 'XVII.' 17
WORD: 'kafla'
WORD: 'laga'
WORD: 'nr.' [('númer', 0, 'hk', 'skst', 'nr.', '-')]
NUMBER: '10' (10, None, None)
PUNCTUATION: '/' (4, '/')
YEAR: '2007' 2007
WORD: 'þann'
DATEABS: '3. janúar 2010' (2010, 1, 3)
PUNCTUATION: '.' (3, '.')
END SENT: '-'Note the following:Sentences are delimited byTOK.S_BEGINandTOK.S_ENDtokens.Composite words, such asstjórnskipunar- og eftirlitsnefndar,
are coalesced into one token.Well-known abbreviations are recognized and their full expansion
is available in thetoken.valfield.Ordinal numbers (3., XVII.) are recognized and their value (3, 17)
is available in thetoken.valfield.Dates, years and times, both absolute and relative, are recognized and
the respective year, month, day, hour, minute and second
values are included as a tuple intoken.val.Numbers, both integer and real, are recognized and their value
is available in thetoken.valfield.Further details of how Tokenizer processes text can be inferred from thetest modulein the project’sGitHub repository.Thetokenize()functionTo deep-tokenize a text string, calltokenizer.tokenize(text, **options).
Thetextparameter can be a string, or an iterable that yields strings
(such as a text file object).The function returns a Pythongeneratorof token objects.
Each token object is a simplenamedtuplewith three
fields:(kind, txt, val)(further documented below).Thetokenizer.tokenize()function is typically called in aforloop:importtokenizerfortokenintokenizer.tokenize(mystring):kind,txt,val=tokenifkind==tokenizer.TOK.WORD:# Do something with word tokenspasselse:# Do something elsepassAlternatively, create a token list from the returned generator:token_list = list(tokenizer.tokenize(mystring))Thesplit_into_sentences()functionTo shallow-tokenize a text string, calltokenizer.split_into_sentences(text_or_gen, **options).
Thetext_or_genparameter can be a string, or an iterable that yields
strings (such as a text file object).This function returns a Pythongeneratorof strings, yielding a string
for each sentence in the input. Within a sentence, the tokens are
separated by spaces.You can pass the optionnormalize=Trueto the function if you want
the normalized form of punctuation tokens. Normalization outputs
Icelandic single and double quotes („these“) instead of English-style
ones (“these”), converts three-dot ellipsis … to single character
ellipsis …, and casts en-dashes – and em-dashes — to regular hyphens.Thetokenizer.split_into_sentences()function is typically called
in aforloop:importtokenizerwithopen("example.txt","r",encoding="utf-8")asf:# You can pass a file object directly to split_into_sentences()forsentenceintokenizer.split_into_sentences(f):# sentence is a string of space-separated tokenstokens=sentence.split()# Now, tokens is a list of strings, one for each tokenfortintokens:# Do something with the token tpassThecorrect_spaces()functionThetokenizer.correct_spaces(text)function returns a string after
splitting it up and re-joining it with correct whitespace around
punctuation tokens. Example:>>> import tokenizer
>>> tokenizer.correct_spaces(
... "Frétt \n  dagsins:Jón\t ,Friðgeir og Páll ! 100  /  2  =   50"
... )
'Frétt dagsins: Jón, Friðgeir og Páll! 100/2 = 50'Thedetokenize()functionThetokenizer.detokenize(tokens, normalize=False)function
takes an iterable of token objects and returns a corresponding, correctly
spaced text string, composed from the tokens’ text. If thenormalizeparameter is set toTrue,
the function uses the normalized form of any punctuation tokens, such
as proper Icelandic single and double quotes instead of English-type
quotes. Example:>>> import tokenizer
>>> toklist = list(tokenizer.tokenize("Hann sagði: „Þú ert ágæt!“."))
>>> tokenizer.detokenize(toklist, normalize=True)
'Hann sagði: „Þú ert ágæt!“.'Thenormalized_text()functionThetokenizer.normalized_text(token)function
returns the normalized text for a token. This means that the original
token text is returned except for certain punctuation tokens, where a
normalized form is returned instead. Specifically, English-type quotes
are converted to Icelandic ones, and en- and em-dashes are converted
to regular hyphens.Thetext_from_tokens()functionThetokenizer.text_from_tokens(tokens)function
returns a concatenation of the text contents of the given token list,
with spaces between tokens. Example:>>> import tokenizer
>>> toklist = list(tokenizer.tokenize("Hann sagði: \"Þú ert ágæt!\"."))
>>> tokenizer.text_from_tokens(toklist)
'Hann sagði : " Þú ert ágæt ! " .'Thenormalized_text_from_tokens()functionThetokenizer.normalized_text_from_tokens(tokens)function
returns a concatenation of the normalized text contents of the given
token list, with spaces between tokens. Example (note the double quotes):>>> import tokenizer
>>> toklist = list(tokenizer.tokenize("Hann sagði: \"Þú ert ágæt!\"."))
>>> tokenizer.normalized_text_from_tokens(toklist)
'Hann sagði : „ Þú ert ágæt ! “ .'Tokenization optionsYou can optionally pass one or more of the following options as
keyword parameters to thetokenize()andsplit_into_sentences()functions:convert_numbers=[bool]Setting this option toTruecauses the tokenizer to convert numbers
and amounts with
English-style decimal points (.) and thousands separators (,)
to Icelandic format, where the decimal separator is a comma (,)
and the thousands separator is a period (.).$1,234.56is thus
converted to a token whose text is$1.234,56.The default value for theconvert_numbersoption isFalse.Note that in versions of Tokenizer prior to 1.4,convert_numberswasTrue.convert_measurements=[bool]Setting this option toTruecauses the tokenizer to convert
degrees Kelvin, Celsius and Fahrenheit to a regularized form, i.e.200° Cbecomes200 °C.The default value for theconvert_measurementsoption isFalse.replace_composite_glyphs=[bool]Setting this option toFalsedisables the automatic replacement
of composite Unicode glyphs with their corresponding Icelandic characters.
By default, the tokenizer combines vowels with the Unicode
COMBINING ACUTE ACCENT and COMBINING DIAERESIS glyphs to form single
character code points, such as ‘á’ and ‘ö’.The default value for thereplace_composite_glyphsoption isTrue.replace_html_escapes=[bool]Setting this option toTruecauses the tokenizer to replace common
HTML escaped character codes, such as&aacute;with the character being
escaped, such asá. Note that&shy;(soft hyphen) is replaced by
an empty string, and&nbsp;is replaced by a normal space.
The ligatures&filig;and&fllig;are replaced byfiandfl,
respectively.The default value for thereplace_html_escapesoption isFalse.handle_kludgy_ordinals=[value]This options controls the way Tokenizer handles ‘kludgy’ ordinals, such as1sti,4ðu, or2ja. By default, such ordinals are returned unmodified
(‘passed through’) as word tokens (TOK.WORD).
However, this can be modified as follows:tokenizer.KLUDGY_ORDINALS_MODIFY: Kludgy ordinals are corrected
to become ‘proper’ word tokens, i.e.1stibecomesfyrstiand2jabecomestveggja.tokenizer.KLUDGY_ORDINALS_TRANSLATE: Kludgy ordinals that represent
proper ordinal numbers are translated to ordinal tokens (TOK.ORDINAL),
with their original text and their ordinal value.1stithus
becomes aTOK.ORDINALtoken with a value of 1, and3jabecomes
aTOK.ORDINALwith a value of 3.tokenizer.KLUDGY_ORDINALS_PASS_THROUGHis the default value of
the option. It causes kludgy ordinals to be returned unmodified as
word tokens.Note that versions of Tokenizer prior to 1.4 behaved as ifhandle_kludgy_ordinalswere set totokenizer.KLUDGY_ORDINALS_TRANSLATE.The token objectEach token is an instance of the classTokthat has three main properties:kind,txtandval.ThekindpropertyThekindproperty contains one of the following integer constants,
defined within theTOKclass:ConstantValueExplanationExamplesPUNCTUATION1Punctuation. ! ; % &TIME2Time (h, m, s)11:35:40kl. 7:05klukkan 23:35DATE *3Date (y, m, d)[Unused, see DATEABS and
DATEREL]YEAR4Yearárið 874 e.Kr.196544 f.Kr.NUMBER5Number1001.9651.965,341,965.342⅞WORD6Wordkattaeftirlithunda- og kattaeftirlitTELNO7Telephone number5254764699-4244410 4000PERCENT8Percentage78%URL9URLhttps://greynir.ishttp://tiny.cc/28695yORDINAL10Ordinal number30.XVIII.TIMESTAMP *11Timestamp[Unused, see
TIMESTAMPABS and
TIMESTAMPREL]CURRENCY *12Currency name[Unused]AMOUNT13Amount€2.345,67750 þús.kr.2,7 mrð. USDkr. 9.900EUR 200PERSON *14Person name[Unused]EMAIL15E-mailfake@news.isENTITY *16Named entity[Unused]UNKNOWN17Unknown tokenDATEABS18Absolute date30. desember 196530/12/19651965-12-301965/12/30DATEREL19Relative date15. mars15/315.3.mars 1911TIMESTAMPABS20Absolute timestamp30. desember 1965 11:341965-12-30 kl. 13:00TIMESTAMPREL21Relative timestamp30. desember kl. 13:00MEASUREMENT22Value with a
measurement unit690 MW1.010 hPa220 m²80° CNUMWLETTER23Number followed by
a single letter14a7BDOMAIN24Domain namegreynir.isReddit.comwww.wikipedia.orgHASHTAG25Hashtag#MeToo#12stigMOLECULE26Molecular formulaH2SO4CO2SSN27Social security
number (kennitala)591213-1480USERNAME28Twitter user handle@username_123SERIALNUMBER29Serial number394-538812-345-6789COMPANY *30Company name[Unused]S_BEGIN11001Start of sentenceS_END11002End of sentence(*) The token types marked with an asterisk are reserved for the Greynir package
and not currently returned by the tokenizer.To obtain a descriptive text for a token kind, useTOK.descr[token.kind](see example above).ThetxtpropertyThetxtproperty contains the original source text for the token,
with the following exceptions:All contiguous whitespace (spaces, tabs, newlines) is coalesced
into single spaces (" ") within thetxtstring. A date
token that is parsed from a source text of"29.  \n   janúar"thus has atxtof"29. janúar".Tokenizer automatically merges UnicodeCOMBINING ACUTE ACCENT(code point 769) andCOMBINING DIAERESIS(code point 776)
with vowels to form single code points for the Icelandic letters
á, é, í, ó, ú, ý and ö, in both lower and upper case. (This behavior
can be disabled; see thereplace_composite_glyphsoption described
above.)If the appropriate options are specified (see above), it converts
kludgy ordinals (3ja) to proper ones (þriðja), and English-style
thousand and decimal separators to Icelandic ones
(10,345.67becomes10.345,67).If thereplace_html_escapesoption is set, Tokenizer replaces
HTML-style escapes (&aacute;) with the characters
being escaped (á).ThevalpropertyThevalproperty contains auxiliary information, corresponding to
the token kind, as follows:ForTOK.PUNCTUATION, thevalfield contains a tuple with
two items:(whitespace, normalform). The first item (token.val[0])
specifies the whitespace normally found around the symbol in question,
as an integer:TP_LEFT = 1   # Whitespace to the left
TP_CENTER = 2 # Whitespace to the left and right
TP_RIGHT = 3  # Whitespace to the right
TP_NONE = 4   # No whitespaceThe second item (token.val[1]) contains a normalized representation of the
punctuation. For instance, various forms of single and double
quotes are represented as Icelandic ones (i.e. „these“ or ‚these‘) in
normalized form, and ellipsis (”…”) are represented as the single
character “…”.ForTOK.TIME, thevalfield contains an(hour, minute, second)tuple.ForTOK.DATEABS, thevalfield contains a(year, month, day)tuple (all 1-based).ForTOK.DATEREL, thevalfield contains a(year, month, day)tuple (all 1-based),
except that a least one of the tuple fields is missing and set to 0.
Example:3. júníbecomesTOK.DATERELwith the fields(0, 6, 3)as the year is missing.ForTOK.YEAR, thevalfield contains the year as an integer.
A negative number indicates that the year is BCE (fyrir Krist),
specified with the suffixf.Kr.(e.g.árið 33 f.Kr.).ForTOK.NUMBER, thevalfield contains a tuple(number, None, None).
(The two empty fields are included for compatibility with Greynir.)ForTOK.WORD, thevalfield contains the full expansion
of an abbreviation, as a list containing a single tuple, orNoneif the word is not abbreviated.ForTOK.PERCENT, thevalfield contains a tuple
of(percentage, None, None).ForTOK.ORDINAL, thevalfield contains the ordinal value
as an integer. The original ordinal may be a decimal number
or a Roman numeral.ForTOK.TIMESTAMP, thevalfield contains
a(year, month, day, hour, minute, second)tuple.ForTOK.AMOUNT, thevalfield contains
an(amount, currency, None, None)tuple. The amount is a float, and
the currency is an ISO currency code, e.g.USDfor dollars ($ sign),EURfor euros (€ sign) orISKfor Icelandic króna
(kr.abbreviation). (The two empty fields are included for
compatibility with Greynir.)ForTOK.MEASUREMENT, thevalfield contains a(unit, value)tuple, whereunitis a base SI unit (such asg,m,m²,s,W,Hz,Kfor temperature in Kelvin).ForTOK.TELNO, thevalfield contains a tuple:(number, cc)where the first item is the phone number
in a normalizedNNN-NNNNformat, i.e. always including a hyphen,
and the second item is the country code, eventually prefixed by+.
The country code defaults to354(Iceland).AbbreviationsAbbreviations recognized by Tokenizer are defined in theAbbrev.conffile, found in thesrc/tokenizer/directory. This is a text file with
abbreviations, their definitions and explanatory comments.When an abbreviation is encountered, it is recognized as a word token
(i.e. having itskindfield equal toTOK.WORD).
Its expansion(s) are included in the token’svalfield as a list containing tuples of the format(ordmynd, utg, ordfl, fl, stofn, beyging).
An example iso.s.frv., which results in avalfield equal to[('ogsvo framvegis', 0, 'ao', 'frasi','o.s.frv.','-')].The tuple format is designed to be compatible with theDatabase of Icelandic Morphology(DIM),Beygingarlýsing íslensks nútímamáls, i.e. the so-calledSigrúnarsnið.Development installationTo install Tokenizer in development mode, where you can easily
modify the source files (assuming you havegitavailable):$gitclonehttps://github.com/mideind/Tokenizer$cdTokenizer$# [ Activate your virtualenv here, if you have one ]$pipinstall-e.Test suiteTokenizer comes with a large test suite.
The filetest/test_tokenizer.pycontains built-in tests that
run underpytest.To run the built-in tests, installpytest,cdto yourTokenizersubdirectory (and optionally
activate your virtualenv), then run:$python-mpytestThe filetest/toktest_large.txtcontains a test set of 13,075 lines.
The lines test sentence detection, token detection and token classification.
For analysis,test/toktest_large_gold_perfect.txtcontains
the expected output of a perfect shallow tokenization, andtest/toktest_large_gold_acceptable.txtcontains the current output of the
shallow tokenization.The filetest/Overview.txt(only in Icelandic) contains a description
of the test set, including line numbers for each part in bothtest/toktest_large.txtandtest/toktest_large_gold_acceptable.txt,
and a tag describing what is being tested in each part.It also contains a description of a perfect shallow tokenization for each part,
acceptable tokenization and the current behaviour.
As such, the description is an analysis of which edge cases the tokenizer
can handle and which it can not.To test the tokenizer on the large test set the following needs to be typed
in the command line:$tokenizetest/toktest_large.txttest/toktest_large_out.txtTo compare it to the acceptable behaviour:$difftest/toktest_large_out.txttest/toktest_large_gold_acceptable.txt>diff.txtThe filetest/toktest_normal.txtcontains a running text from recent
news articles, containing no edge cases. The gold standard for that file
can be found in the filetest/toktest_normal_gold_expected.txt.ChangelogVersion 3.4.3: Various minor fixes. Now requires Python 3.8 or later.Version 3.4.2: Abbreviations and phrases added,META_BEGINtoken added.Version 3.4.1: Improved performance on long input chunks.Version 3.4.0: Improved handling and normalization of punctuation.Version 3.3.2: Internal refactoring; bug fixes in paragraph handling.Version 3.3.1: Fixed bug where opening quotes at the start of paragraphs
were sometimes incorrectly recognized and normalized.Version 3.2.0: Numbers and amounts that consist of word tokens only (‘sex hundruð’)
are now returned as the originalTOK.WORDs (‘sex’ and ‘hundruð’), not as single
coalescedTOK.NUMBER/TOK.AMOUNT/etc. tokens.Version 3.1.2: Changed paragraph markers to[[and]](removing spaces).Version 3.1.1: Minor fixes; added Tok.from_token().Version 3.1.0: Added-oswitch to thetokenizecommand to return original
token text, enabling the tokenizer to run as a sentence splitter only.Version 3.0.0: Added tracking of character offsets for tokens within the
original source text. Added full type annotations. Dropped Python 2.7 support.Version 2.5.0: Added arguments for all tokenizer options to the
command-line tool. Type annotations enhanced.Version 2.4.0: Fixed bug where certain well-known word forms (fá,fær,mín,sá…)
were being interpreted as (wrong) abbreviations. Also fixed bug where certain
abbreviations were being recognized even in uppercase and at the end
of a sentence, for instanceÖrn.Version 2.3.1: Various bug fixes; fixed type annotations for Python 2.7;
the token kindNUMBER WITH LETTERis nowNUMWLETTER.Version 2.3.0: Added thereplace_html_escapesoption to
thetokenize()function.Version 2.2.0: Fixedcorrect_spaces()to handle compounds such asAtvinnu-, nýsköpunar- og ferðamálaráðuneytiðandbensínstöðvar, -dælur og -tankar.Version 2.1.0: Changed handling of periods at end of sentences if they are
a part of an abbreviation. Now, the period is kept attached to the abbreviation,
not split off into a separate period token, as before.Version 2.0.7: AddedTOK.COMPANYtoken type; fixed a few abbreviations;
renamed parametertexttotext_or_genin functions that accept a string
or a string iterator.Version 2.0.6: Fixed handling of abbreviations such asm.v.(miðað við)
that should not start a new sentence even if the following word is capitalized.Version 2.0.5: Fixed bug where single uppercase letters were erroneously
being recognized as abbreviations, causing prepositions such as ‘Í’ and ‘Á’
at the beginning of sentences to be misunderstood in GreynirPackage.Version 2.0.4: Added imperfect abbreviations (amk.,osfrv.); recognizedklukkan hálf tvöas aTOK.TIME.Version 2.0.3: Fixed bug indetokenize()where abbreviations, domains
and e-mails containing periods were wrongly split.Version 2.0.2: Spelled-out day ordinals are no longer included as a part ofTOK.DATERELtokens. Thus,þriðji júníis now aTOK.WORDfollowed by aTOK.DATEREL.3. júnícontinues to be parsed as
a singleTOK.DATEREL.Version 2.0.1: Order of abbreviation meanings within thetoken.valfield
made deterministic; fixed bug in measurement unit handling.Version 2.0.0: Added command line tool; addedsplit_into_sentences()anddetokenize()functions; removedconvert_telnooption;
splitting of coalesced tokens made more robust;
addedTOK.SSN,TOK.MOLECULE,TOK.USERNAMEandTOK.SERIALNUMBERtoken kinds; abbreviations can now have multiple
meanings.Version 1.4.0: Added the**optionsparameter to thetokenize()function, giving control over the handling of numbers,
telephone numbers, and ‘kludgy’ ordinals.Version 1.3.0: AddedTOK.DOMAINandTOK.HASHTAGtoken types;
improved handling of capitalized month nameÁgúst, which is
now recognized when following an ordinal number; improved recognition
of telephone numbers; added abbreviations.Version 1.2.3: Added abbreviations; updated GitHub URLs.Version 1.2.2: Added support for composites with more than two parts, i.e.„dómsmála-, ferðamála-, iðnaðar- og nýsköpunarráðherra“; added support for±sign; added several abbreviations.Version 1.2.1: Fixed bug where the nameÁgústwas recognized
as a month name; Unicode nonbreaking and invisible space characters
are now removed before tokenization.Version 1.2.0: Added support for Unicode fraction characters;
enhanced handing of degrees (°, °C, °F); fixed bug in cubic meter
measurement unit; more abbreviations.Version 1.1.2: Fixed bug in liter (landltr) measurement units.Version 1.1.1: Addedmark_paragraphs()function.Version 1.1.0: All abbreviations inAbbrev.confare now
returned with their meaning in a tuple intoken.val;
handling of ‘mbl.is’ fixed.Version 1.0.9: Added abbreviation ‘MAST’; harmonized copyright headers.Version 1.0.8: Bug fixes inDATEREL,MEASUREMENTandNUMWLETTERtoken handling; added ‘kWst’ and ‘MWst’ measurement units; blackened.Version 1.0.7: AddedTOK.NUMWLETTERtoken type.Version 1.0.6: Automatic merging of UnicodeCOMBINING ACUTE ACCENTandCOMBINING DIAERESIScode points with vowels.Version 1.0.5: Date/time and amount tokens coalesced to a further extent.Version 1.0.4: AddedTOK.DATEABS,TOK.TIMESTAMPABS,TOK.MEASUREMENT.