Package: abydos
AbydosCI & Test StatusCode QualityDependenciesLocal AnalysisUsageContributionPyPIconda-forgeAbydos NLP/IR libraryCopyright 2014-2020 by Christopher C. LittleAbydos is a library of phonetic algorithms, string distance measures & metrics,
stemmers, and string fingerprinters including:Phonetic algorithmsRobert C. Russell’s IndexAmerican SoundexRefined SoundexDaitch-Mokotoff SoundexKölner PhonetikNYSIISMatch Rating AlgorithmMetaphoneDouble MetaphoneCaverphoneAlpha Search Inquiry SystemFuzzy SoundexPhonexPhonemPhonixSfinxBisphonetStandardized Phonetic Frequency CodeStatistics CanadaLeinRoger RootOxford Name Compression Algorithm (ONCA)Eudex phonetic hashHaase PhonetikReth-Schek PhonetikFONEMParmar-KumbharanaDavidson’s Consonant CodeSoundDPSHP Soundex/Viewex Codingan early version of Henry CodeNorphoneDolby CodePhonetic SpanishSpanish MetaphoneMetaSoundexSoundexBRNRL English-to-phonemeBeider-Morse Phonetic MatchingString distance metricsLevenshtein distanceOptimal String Alignment distanceLevenshtein-Damerau distanceHamming distanceTversky indexSørensen–Dice coefficient & distanceJaccard similarity coefficient & distanceoverlap similarity & distanceTanimoto coefficient & distanceMinkowski distance & similarityManhattan distance & similarityEuclidean distance & similarityChebyshev distancecosine similarity & distanceJaro distanceJaro-Winkler distance (incl. the strcmp95 algorithm variant)Longest common substringRatcliff-Obershelp similarity & distanceMatch Rating Algorithm similarityNormalized Compression Distance (NCD) & similarityMonge-Elkan similarity & distanceMatrix similarityNeedleman-Wunsch scoreSmith-Waterman scoreGotoh scoreLength similarityPrefix, Suffix, and Identity similarity & distanceModified Language-Independent Product Name Search (MLIPNS) similarity &
distanceBag distanceEditex distanceEudex distancesSift4 distanceBaystat distance & similarityTypo distanceIndel distanceSynonameStemmersthe Lovins stemmerthe Porter and Porter2 (Snowball English) stemmersSnowball stemmers for German, Dutch, Norwegian, Swedish, and DanishCLEF German, German plus, and Swedish stemmersCaumann’s German stemmerUEA-Lite StemmerPaice-Husk StemmerSchinke Latin stemmerS stemmerString Fingerprintsstring fingerprintq-gram fingerprintphonetic fingerprintPollock & Zomora’s skeleton keyPollock & Zomora’s omission keyCisłak & Grabowski’s occurrence fingerprintCisłak & Grabowski’s occurrence halved fingerprintCisłak & Grabowski’s count fingerprintCisłak & Grabowski’s position fingerprintSynoname ToolcodeInstallationRequired libraries:NumPydeprecationOptional libraries (all available on PyPI, some available on conda or
conda-forge):SyllabiPyNLTKPyLZSSpaqTo install Abydos (master) from Github source:git clone https://github.com/chrislit/abydos.git --recursive
cd abydos
python setup installIf your default python command calls Python 2.7 but you want to install for
Python 3, you may instead need to call:python3 setup installTo install Abydos (latest release) from PyPI using pip:pip install abydosTo install fromconda-forge:conda install abydosIt should run on Python 3.5-3.8.Testing & ContributingTo run the whole test-suite just call tox:toxThe tox setup has the following environments: black, py37, doctest,
regression, fuzz, pylint, pydocstyle, flake8, doc8, docs, sloccount, badges, &
build. So if you only want to generate documentation (in HTML, EPUB, & PDF
formats), just call:tox -e docsIn order to only run & generate Flake8 reports, call:tox -e flake8Contributions such as bug reports, PRs, suggestions, desired new features, etc.
are welcome through GithubIssues&Pull requests.Release History0.5.0 (2020-01-10)ecgtheowdoi:10.5281/zenodo.3603514Changes:Support for Python 2.7 was removed.0.4.1 (2020-01-07)distant dietrichdoi:10.5281/zenodo.3600548Changes:Support for Python 3.4 was removed. (3.4 reached end-of-life on March 18,
2019)Fuzzy intersections were corrected to avoid over-counting partial
intersection instances.Levenshtein can now return an optimal alignmentAdded the following distance measures:Indice de Similitude-Guth (ISG)INClusion ProgrammeGuthVictorian Panel Study (VPS) scoreLIG3 similarityDiscounted LevenshteinRelaxed HammingString subsequence kernel (SSK) similarityPhonetic edit distanceHenderson-Heron dissimilarityRaup-Crick similarityMillar’s binomial deviance dissimilarityMorisita similarityHorn-Morisita similarityClark’s coefficient of divergenceChao’s Jaccard similarityChao’s Dice similarityCao’s CY similarity (CYs) and dissimilarity (CYd)Added the following fingerprint classes:Taft’s Consonant codingTaft’s Extract - letter listTaft’s Extract - position & frequencyL.A. County Sheriff’s SystemLibrary of Congres Cutter table encodingAdded the following phonetic algorithms:Ainsworth’s grapheme-to-phonemePHONIC0.4.0 (2019-05-30)dietrichdoi:10.5281/zenodo.3235034Version 0.4.0 focuses on distance measures, adding 211 new measures. Attempts
were made to provide normalized version for measure that did not inherently
range from 0 to 1. The other major focus was the addition of 12 tokenizers, in
service of expanding distance measure options.Changes:Support for Python 3.3 was dropped.Deprecated functions that merely wrap class methods to maintain API
compatibility, for removal in 0.6.0Added methods to ConfusionTable to return:its internal representationfalse negative ratefalse omission ratepositive & negative likelihood ratiosdiagnostic odds ratioerror rateprevalenceJaccard indexD-measurePhi coefficientjoint, actual, & predicted entropiesmutual informationproficiency (uncertainty coefficient)information gain ratiodependencyliftDeprecated f-measure & g-measure from ConfusionTable for removal in
0.6.0Added notes to indicate when functions, classes, & methods were addedAdded the following 12 tokenizers:QSkipgramsCharacterTokenizerRegexpTokenizer, WhitespaceTokenizer, & WordpunctTokenizerCOrVClusterTokenizer, CVClusterTokenizer, & VCClusterTokenizerSonoriPyTokenizer & LegaliPyTokenizerNLTKTokenizerSAPSTokenizerAdded the UnigramCorpus class & a facility for downloading data, such as
pre-processed/trained data, from storage on GitHubAdded the Wåhlin phonetic encodingAdded the following 211 similarity/distance/correlation measures:ALINEAMPLEAnderbergAndres & Marzo’s DeltaAverage LinkageAZZOOBaroni-Urbani & Buser I & IIBatagelj & BrenBaulieu I-XVBenini I & IIBennetBhattacharyyaBI-SIMBLEUBlock LevenshteinBrainerd-RobinsonBraun-BlanquetCanberraChordClementCohen’s KappaColeComplete LinkageConsonni & Todeschini I-VCormode’s LZCovingtonDennisDice Asymmetric I & IIDigbyDispersionDoolittleDunningEyraudFager & McGowanFaithFellegi-SunterFidelityFleissFleiss-Levin-PaikFlexMetricForbes I & IIFossumFuzzyWuzzy Partial StringFuzzyWuzzy Token SetFuzzyWuzzy Token SortGeneralized FleissGilbertGilbert & WellsGini I & IIGoodallGoodman & Kruskal’s LambdaGoodman & Kruskal’s Lambda-rGoodman & Kruskal’s Tau A & BGower & LegendreGuttman’s Lambda A & BGwet’s ACHamannHarris & LaheyHassanatHawkins & DotsonHellingerHiguera & MicoHurlbertIterative SubStringJaccard-NMJensen-ShannonJohnsonKendall’s TauKent & Foster I & IIKoppen I & IIKuder & RichardsonKuhns I-XIIKulczynski I & IILongest Common PrefixLongest Common SuffixLorentzianMaarelMarkingMarking MetricMASIMatusitaMaxwell & PillinerMcConnaugheyMcEwen & MichaelMetaLevenshteinMicheletMinHashMountfordMean Squared ContingencyMutual InformationNCD with LZSSNCD with PAQ9aOzbayPatternPearson’s Chi-SquaredPearson & Heron IIPearson II & IIIPearson’s PhiPeircePositional Q-Gram Dice, Jaccard, & OverlapQ-GramQuantitative Cosine, Dice, & JaccardRees-LevenshteinRobertsRogers & TanimotoRogot & GoldbergRouge-L, -S, -SU, & -WRussell & RaoSAPSScott’s PiShapeShapira & Storer ISift4 ExtendedSingle LinkageSizeSoft CosineSoftTF-IDFSokal & MichenerSokal & Sneath I-VSorgenfreiSteffensenStilesStuart’s TauTarantulaTarwidTetrachoricTF-IDFTichyTulloss’s R, S, T, & UUnigram SubtupleUnknown A-MUpholtWarrens I-VWeighted JaccardWhittakerYates’ Chi-SquaredYJHHRYujian & BoYule’s Q, Q II, & YFour intersection types are now supported for all distance measure that are
based on _TokenDistance. In addition to basic crisp intersections, soft,
fuzzy, and group linkage intersections have been provided.0.3.6 (2018-11-17)classy carldoi:10.5281/zenodo.1490537Changes:Most functions were encapsulated into classes.Each class is broken out into its own file, with test files paralleling
library files.Documentation was converted from Sphinx markup to Numpy style.A tutorial was written for each subpackage.Documentation was cleaned up, with math markup corrections and many
additional links.0.3.5 (2018-10-31)cantankerous carldoi:10.5281/zenodo.1463204Version 0.3.5 focuses on refactoring the whole project. The API itself remains
largely the same as in previous versions, but underlyingly modules have been
split up. Essentially no new features are added (bugfixes aside) in this
version.Changes:Refactored library and tests into smaller modulesBroke compression distances (NCD) out into separate functionsAdopted Black code styleAdded pyproject.toml to use Poetry for packaging (but will continue using
setuptools and setup.py for the present)Minor bug fixes0.3.0 (2018-10-15)carldoi:10.5281/zenodo.1462443Version 0.3.0 focuses on additional phonetic algorithms, but does add numerous
distance measures, fingerprints, and even a few stemmers. Another focus was
getting everything to build again (including docs) and to move to more
standard modern tools (flake8, tox, etc.).Changes:Fixed implementation of Bag distanceUpdated BMPM to version 3.10Fixed Sphinx documentation on readthedocs.orgSplit string fingerprints out of clustering into their own moduleAdded support for q-grams to skip-n charactersNew phonetic algorithms:Statistics CanadaLeinRoger RootOxford Name Compression Algorithm (ONCA)Eudex phonetic hashHaase PhonetikReth-Schek PhonetikFONEMParmar-KumbharanaDavidson’s Consonant CodeSoundDPSHP Soundex/Viewex Codingan early version of Henry CodeNorphoneDolby CodePhonetic SpanishSpanish MetaphoneMetaSoundexSoundexBRNRL English-to-phonemeNew string fingerprints:Cisłak & Grabowski’s occurrence fingerprintCisłak & Grabowski’s occurrence halved fingerprintCisłak & Grabowski’s count fingerprintCisłak & Grabowski’s position fingerprintSynoname ToolcodeNew distance measures:Minkowski distance & similarityManhattan distance & similarityEuclidean distance & similarityChebyshev distance & similarityEudex distancesSift4 distanceBaystat distance & similarityTypo distanceIndel distanceSynonameNew stemmers:UEA-Lite StemmerPaice-Husk StemmerSchinke Latin stemmerS stemmerEliminated ._compat submodule in favor of sixTransitioned from PEP8 to flake8, etc.Phonetic algorithms now consistently use max_length=-1 to indicate that
there should be no length limitAdded example notebooks in binder directory0.2.0 (2015-05-27)bertholdAdded Caumanns’ German stemmerAdded Lovins’ English stemmerUpdated Beider-Morse Phonetic Matching to 3.04Added Sphinx documentation0.1.1 (2015-05-12)albrechtFirst Beta release to PyPIAuthorsChristopher C. Little (@chrislit) <chrisclittle+abydos@gmail.com>