Package: icegrams
OverviewIcegramsis an MIT-licensed Python 3 (>= 3.7) package that encapsulates alarge trigram library for Icelandic. (A trigram is a tuple of
three consecutive words or tokens that appear in real-world text.)14 million unique trigrams and their frequency counts are heavily compressed
using radix tries andquasi-succinct indexesemploying Elias-Fano encoding. This enables the ~43 megabyte compressed trigram file
to be mapped directly into memory, with noex antedecompression, for fast queries
(typically ~10 microseconds per lookup).The Icegrams library is implemented in Python and C/C++, glued together viaCFFI.The trigram storage approach is based on a2017 paper by Pibiri and Venturini,
also referring toOttaviano and Venturini(2014) regarding partitioned Elias-Fano indexes.You can use Icegrams to obtain probabilities (relative frequencies) of
over a million differentunigrams(single words or tokens), or ofbigrams(pairs of two words or tokens), or oftrigrams. You can also
ask it to return the N most likely successors to any unigram or bigram.Icegrams is useful for instance in spelling correction, predictive typing,
to help disabled people write text faster, and for various text generation,
statistics and modelling tasks.The Icegrams trigram corpus is built from the 2017 edition of the
Icelandic Gigaword Corpus
(Risamálheild),
which is collected and maintained byThe Árni Magnússon Institutefor Icelandic Studies. A mixed, manually vetted subset consisting of 157
documents from the corpus was used as the source of the token stream,
yielding over 100 million tokens. Trigrams that only occurred
once or twice in the stream were eliminated before creating the
compressed Icegrams database. The creation process is furtherdescribed here.Example>>> from icegrams import Ngrams
>>> ng = Ngrams()
>>> # Obtain the frequency of the unigram 'Ísland'
>>> ng.freq("Ísland")
42018
>>> # Obtain the probability of the unigram 'Ísland', as a fraction
>>> # of the frequency of all unigrams in the database
>>> ng.prob("Ísland")
0.0003979926900206475
>>> # Obtain the log probability (base e) of the unigram 'Ísland'
>>> ng.logprob("Ísland")
-7.8290769196308005
>>> # Obtain the frequency of the bigram 'Katrín Jakobsdóttir'
>>> ng.freq("Katrín", "Jakobsdóttir")
3517
>>> # Obtain the probability of 'Jakobsdóttir' given 'Katrín'
>>> ng.prob("Katrín", "Jakobsdóttir")
0.23298013245033142
>>> # Obtain the probability of 'Júlíusdóttir' given 'Katrín'
>>> ng.prob("Katrín", "Júlíusdóttir")
0.013642384105960274
>>> # Obtain the frequency of 'velta fyrirtækisins er'
>>> ng.freq("velta", "fyrirtækisins", "er")
4
>>> # adj_freq returns adjusted frequencies, i.e incremented by 1
>>> ng.adj_freq("xxx", "yyy", "zzz")
1
>>> # Obtain the N most likely successors of a given unigram or bigram,
>>> # in descending order by log probability of each successor
>>> ng.succ(10, "stjórnarskrá", "lýðveldisins")
[('Íslands', -1.3708244393477589), ('.', -2.2427905461504567),
    (',', -3.313814878299737), ('og', -3.4920631097060557), ('sem', -4.566577846795106),
    ('er', -4.720728526622363), ('að', -4.807739903611993), ('um', -5.0084105990741445),
    ('en', -5.0084105990741445), ('á', -5.25972502735505)]ReferenceInitializing IcegramsAfter installing theicegramspackage, use the following code to
import it and initialize an instance of theNgramsclass:from icegrams import Ngrams
ng = Ngrams()Now you can use thenginstance to query for unigram, bigram
and trigram frequencies and probabilities.The Ngrams class__init__(self)Initializes theNgramsinstance.freq(self, *args)->intReturns the frequency of a unigram, bigram or trigram.str[] *argsA parameter sequence of consecutive unigrams
to query the frequency for.returnsAn integer with the frequency of the unigram,
bigram or trigram.To query for the frequency of a unigram in the text, callng.freq("unigram1"). This returns the number of times that
the unigram appears in the database. The unigram is
queried as-is, i.e. with no string stripping or lowercasing.To query for the frequency of a bigram in the text, callng.freq("unigram1","unigram2").To query for the frequency of a trigram in the text, callng.freq("unigram1","unigram2", "unigram3").If you pass more than 3 arguments tong.freq(), only the
last 3 are significant, and the query will be treated
as a trigram query.Examples:>>>> ng.freq("stjórnarskrá")
2973
>>>> ng.freq("stjórnarskrá", "lýðveldisins")
39
>>>> ng.freq("stjórnarskrá", "lýðveldisins", "Íslands")
12
>>>> ng.freq("xxx", "yyy", "zzz")
0adj_freq(self, *args)->intReturns the adjusted frequency of a unigram, bigram or trigram.str[] *argsA parameter sequence of consecutive unigrams
to query the frequency for.returnsAn integer with the adjusted frequency of the unigram,
bigram or trigram. The adjusted frequency is the actual
frequency plus 1. The method thus never returns 0.To query for the frequency of a unigram in the text, callng.adj_freq("unigram1"). This returns the number of times that
the unigram appears in the database, plus 1. The unigram is
queried as-is, i.e. with no string stripping or lowercasing.To query for the frequency of a bigram in the text, callng.adj_freq("unigram1","unigram2").To query for the frequency of a trigram in the text, callng.adj_freq("unigram1","unigram2", "unigram3").If you pass more than 3 arguments tong.adj_freq(), only the
last 3 are significant, and the query will be treated
as a trigram query.Examples:>>>> ng.adj_freq("stjórnarskrá")
2974
>>>> ng.adj_freq("stjórnarskrá", "lýðveldisins")
40
>>>> ng.adj_freq("stjórnarskrá", "lýðveldisins", "Íslands")
13
>>>> ng.adj_freq("xxx", "yyy", "zzz")
1prob(self, *args)->floatReturns the probability of a unigram, bigram or trigram.str[] *argsA parameter sequence of consecutive unigrams
to query the probability for.returnsA float with the probability of the given unigram,
bigram or trigram.The probability of aunigramis
the frequency of the unigram divided by the sum of the
frequencies of all unigrams in the database.The probability of abigram(u1, u2)is the frequency
of the bigram divided by the frequency of the unigramu1,
i.e. how likelyu2is to succeedu1.The probability of a trigram(u1, u2, u3)is the frequency
of the trigram divided by the frequency of the bigram(u1, u2),
i.e. how likelyu3is to succeedu1 u2.If you pass more than 3 arguments tong.prob(), only the
last 3 are significant, and the query will be treated
as a trigram probability query.Examples:>>>> ng.prob("stjórnarskrá")
2.8168929772755334e-05
>>>> ng.prob("stjórnarskrá", "lýðveldisins")
0.01344989912575655
>>>> ng.prob("stjórnarskrá", "lýðveldisins", "Íslands")
0.325logprob(self, *args)->floatReturns the log probability of a unigram, bigram or trigram.str[] *argsA parameter sequence of consecutive unigrams
to query the log probability for.returnsA float with the natural logarithm (basee) of the
probability of the given unigram, bigram or trigram.The probability of aunigramis
the adjusted frequency of the unigram divided by the sum of the
frequencies of all unigrams in the database.The probability of abigram(u1, u2)is the adjusted frequency
of the bigram divided by the adjusted frequency of the unigramu1,
i.e. how likelyu2is to succeedu1.The probability of a trigram(u1, u2, u3)is the adjusted frequency
of the trigram divided by the adjusted frequency of the bigram(u1, u2),
i.e. how likelyu3is to succeedu1 u2.If you pass more than 3 arguments tong.logprob(), only the
last 3 are significant, and the query will be treated
as a trigram probability query.Examples:>>>> ng.logprob("stjórnarskrá")
-10.477290968535172
>>>> ng.logprob("stjórnarskrá", "lýðveldisins")
-4.308783672906165
>>>> ng.logprob("stjórnarskrá", "lýðveldisins", "Íslands")
-1.1239300966523995succ(self, n, *args)->list[tuple]Returns theNmost probable successors of a unigram or bigram.int nA positive integer specifying how many successors,
at a maximum, should be returned.str[] *argsOne or two string parameters containing the
unigram or bigram to query the successors for.returnsA list of tuples of (successor unigram, log probability),
in descending order of probability.If you pass more than 2 string arguments tong.succ(), only the
last 2 are significant, and the query will be treated
as a bigram successor query.Examples:>>>> ng.succ(2, "stjórnarskrá")
[('.', -1.8259625296091855), ('landsins', -2.223111581475692)]
>>>> ng.succ(2, "stjórnarskrá", "lýðveldisins")
[('Íslands', -1.1239300966523995), ('og', -1.3862943611198904)]
>>>> # The following is equivalent to ng.succ(2, "lýðveldisins", "Íslands")
>>>> ng.succ(2, "stjórnarskrá", "lýðveldisins", "Íslands")
[('.', -1.3862943611198908), (',', -1.6545583477145702)]NotesIcegrams is built with a sliding window over the source text. This means that
a sentence such as"Maðurinn borðaði ísinn."results in the following
trigrams being added to the database:("", "", "Maðurinn")
("", "Maðurinn", "borðaði")
("Maðurinn", "borðaði", "ísinn")
("borðaði", "ísinn", ".")
("ísinn", ".", "")
(".", "", "")The same sliding window strategy is applied for bigrams, so the following
bigrams would be recorded for the same sentence:("", "Maðurinn")
("Maðurinn", "borðaði")
("borðaði", "ísinn")
("ísinn", ".")
(".", "")You can thus obtain the N unigrams that most often start
a sentence by asking forng.succ(N, "").And, of course, four unigrams are also added, one for each token in the
sentence.The tokenization of the source text into unigrams is done with theTokenizer packageand
uses the rules documented there. Importantly, tokens other than words,
abbreviations, entity names, person names and punctuation arereplaced by placeholders. This means that all numbers are represented by the token[NUMBER], amounts by[AMOUNT], dates by[DATEABS]and[DATEREL],
e-mail addresses by[EMAIL], etc. For the complete mapping of token types
to placeholder strings, see thedocumentation for the Tokenizer package.PrerequisitesThis package runs on CPython 3.6 or newer, and on PyPy 3.6 or newer. It
has been tested on Linux (gcc on x86-64 and ARMhf), MacOS (clang) and
Windows (MSVC).If a binary wheel package isn’t available onPyPIfor your system, you may need to have thepython3-devpackage
(or its Windows equivalent) installed on your system to set up
Icegrams successfully. This is because a source distribution
install requires a C++ compiler and linker:# Debian or Ubuntu:
sudo apt-get install python3-devInstallationTo install this package:$ pip install icegramsIf you want to be able to edit the source, do like so (assuming you havegitinstalled):$ git clone https://github.com/mideind/Icegrams
$ cd Icegrams
$ # [ Activate your virtualenv here if you have one ]
$ python setup.py developThe package source code is now in./src/icegrams.TestsTo run the built-in tests, installpytest,cdto yourIcegramssubdirectory (and optionally activate your
virtualenv), then run:$ python -m pytestChangelogVersion 1.1.2: Minor bug fixes. Cross-platform wheels provided. Now requires Python 3.7+. (2022-12-14)Version 1.1.0: Python 3.5 support dropped; macOS builds fixed; PyPy wheels
generatedVersion 1.0.0: New trigram database sourced from the Icelandic Gigaword Corpus
(Risamálheild) with improved tokenization. Replaced GNU GPLv3 with MIT license.Version 0.6.0: Python type annotations addedVersion 0.5.0: Trigrams corpus has been spell-checkedCopyright and licensingGreynir iscopyright © 2022byMiðeind ehf..
The original author of this software isVilhjálmur Þorsteinsson.This software is licensed under theMIT License:Permission is hereby granted, free of charge, to any personobtaining a copy of this software and associated documentationfiles (the “Software”), to deal in the Software without restriction,including without limitation the rights to use, copy, modify, merge,publish, distribute, sublicense, and/or sell copies of the Software,and to permit persons to whom the Software is furnished to do so,subject to the following conditions:The above copyright notice and this permission notice shall beincluded in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANYCLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THESOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.