Package: openiti
openitiThis is a first attempt to create a Python library that combines all often-used code in the OpenITI project.
Full documentation and deescription can be found here:https://openiti.readthedocs.io/Installationpip install OpenITIAlternatively, you might need to usepip3 install OpenITIorpython -m pip install OpenITI.Change log:v.0.1.5.10:helper.ara:Arabic-Indic digits and Extended Arabic-Indic digits are removed from thear_charsliststring, and put into a new liststring:ar_nums.
This has the effect that numbers written written with these characters
are not considered Arabic tokens anymore
(just like numbers written with Western Arabic numerals).A large number of Greek, Coptic, Syriac and Latin characters are added to theallowed_charsstringlist, which means they should not be removed
from texts before putting them into the corpus.helper.funcs.text_cleaner: this function, which removes all non-word-characters,
numbers and Latin-script characters from the texts, now useshelper.ara.transcription_charsinstead of[A-z]to define Latin-script letters.
This means it will now also remove common transcription letters
(ā, ḥ, ...) instead of only ASCII letters.helper.rgx:add a list of Islamicate language codesadd regex for author (author_uri), book (book_uri) and version URIs
(version_uri)fix the page number related regexes to include PageBeg and PageEnd tags,
and folio numbers that end with lower-case "a" or "b"adapt thesection_tagregex to include this new flavour:### |5|(which is the same as a section tag with five pipes)add anall_tagsregex that can be used to remove all OpenITI mARkdown tagsadd anhtml_tagsregex that can be used to find html tagshelper.uri:build_pth: take into account the different repo name formats
for Arabic and other languages (Arabic: 0025AH, 0050AH, ...;
Persian: PER0025AH, PER0050AH, ...; Urdu: URD0025AH, ...)change_uri: add anon_25Y_folderargument. Set this to True
if you want to use the function for folders that do not have
subfolders for each 25-year period.add_character_count: idemmove_yml: idemmake_folder: idemmove_to_new_uri_pth: idemcheck_yml_files: return list of paths to yml files where the checks failed
instead ofNonenew_books.add.add_books: implementnon_25Y_folderargument in all functions
(see above inhelper.uri)new_books.convert.epub_converter_hindawi: deal with possibility of
unavailable metadatanew_books.convert.helper.html2md: improve named entity taggingnew_books.convert.helper.html2md_LAL: various improvements for
formatting Library of Arabic Literature XML filesopeniti.new_books.convert.tei_converter_LAL: idemnew_books.convert.tei_converter_Wuerzburg: small post-processing tweaks.v.0.1.5.9:helper.yml: fix bug: passreflowparameter in readYML function to ymlToDichelper.funcs: Add functions:read_header: read header of an OpenITI file (local path / URL)read_text: read text of an OpenITI file (local path / URL)get_page_number: get the page number of a token based on its offsetget_semantic_tag_elements: extract semantic tags (like @TOP, @PER) from an OpenITI textfind_section_title: get the section title of any location inside a textget_sections: get a list of all sections in an OpenITI texthelper.ara:Add a whitelist of characters that are allowed in OpenITI texts, with support for Hebrew and Cyrillic characters.Add new characters (subscript alef, inverted damma, quranic sukun, small high madda) to thenoisevariablenew_books.convert.helper.html2md: Add underscore to allowed characters in named entity tags and fix named entity countnew_books.convert.tei_converter_LAL: add a new converter for TEI texts from the Library of Arabic Literaturev.0.1.5.8:helper.uri: when a book URI changes, also change references to it in related books.helper.yml: add functions to check completeness of yml filesv.0.1.5.7:helper.uri: fix bug in the extension looping process of thecheck_token_countfunctionv.0.1.5.6:helper.funcs: addnatural_sortfunction to sort a list of strings
that include numbers in natural order (e.g., ["1", "2", "10"] instead of ["1", "10", "2"] )helper.uri: give files without extension priority over files with ".inProgress"
extension in deciding which text file to use to count characters and tokens
for a specific version yml file.new_books.convert.epub_converter_masaha.py: remove superfluous backslash in EDITOR tagnew_books.convert.helper.html2md.py: fix bug in token countopeniti.new_books.convert.helper.html2md_eShia.py: fix bug in footnote conversionopeniti.new_books.convert.html_converter_eShia.py: improve eShia conversionAdd converters for Ghbook, Ghaemiyeh and Rafed files.v.0.1.5.5:helper.uri: add support for flat folders.v.0.1.5.4: bug fixhelper.yml: fix remaining bugs with long lines.helper.uri: fix bugs incheck_yml_filefunction.v.0.1.5.3: bug fixhelper.yml: make sure that yml keys always contain a hashtag.v.0.1.5.2: bug fixhelper.uri: Remove test that blocked the script.v.0.1.5.1: bug fixhelper.yml: fix bug related to long lines in thedicToYMLfunction.v.0.1.5:helper.yml: addfix_broken_ymlfunction to fix yml files
that are unreadable due to indentation problems
(or keys that don't end with a colon)helper.uri: rewrite thecheck_yml_filesfunction to fix
a bug in the character count and add additional checks.helper.funcs: allow more than oneyml_typein the functionget_all_yml_files_in_folder.helper.ara:Add missing EXTENDED ARABIC-INDIC DIGITS characters 67890Add tokenize functionFix typos innormalize_perdoctestnew_books.convert: add converter for Masāḥa Ḥurra epub files
(epub_converter_masaha.py, with helper filehtml2md_masaha.py)new_books.convert.epub_converter_generic.py: implement overwrite
option for (dis)allowing overwriting existing converted files.v.0.1.4:helper.templates: replace the multiple book relations fields in the
book yml file with a single field,#40#BOOK#RELATED##:.helper.yml: make not rearranging lines ("reflowing") in yml files the default,
and change the default line length to 80.helper.funcs: add aget_all_yml_files_in_folder, analogous to the existingget_all_text_files_in_folderfunctionv.0.1.3:new_books.convert: add converters for ALCorpus and Ptolemaeus textsnew_books.convert.helper.html2md: tweaks to import of options + small tweakshelper.ara: Stop ar_cnt_file from raising exception if book misses splitter; instead, print warninghelper.funcs:fix bug inget_all_text_files_in_folderfunction: missing periods in regex.improve missing splitter messageuseara.normalize_ara_lightfunction instead ofara.normalize_ara_extra_lightintext_cleanerfunctionhelper.uri:make it possible to pass a specificversion_fpto thecheck_token_countfunction; before, that function generated that path from the URI, but this created problems when files were not stored in the standard OpenITI folders.addfind_latestparameter incheck_token_countfunction; if ``False, the function will count the tokens in the specificversion_fp` provided; if `True, the script will count tokens in the file with the most advanced extension (.mARkdown > .completed > .inProgress > [no extension])helper.yml:make it possible to pass a specificyml_fpto theymlToDicfunction, so that the script can print the path (if provided) for signalling empty yml files.Include possibility that yml key ends with more than one colonreadYML: add exception message when yml file could not be read.v.0.1.2:openiti.helper.funcs: Fixed bug in report_missing_numbers function.openiti.new_books.convert: added ShamAY converter and small updates to
other shamela converters.v.0.1.1:openiti.helper.funcs: Added get_all_text_files_in_folder() generatoropeniti.helper.uri: Fix bug innew_ymlfunction (URI used to have ".yml" in it)openiti.new_books.convert.shamela_converter.py: Improved formatting of the text and notes and added support for shamela collections in which the .mdb files contain more than one book.openiti.new_books.convert.tei_converter_Thielen,new_books/convert/helper/html2md_Thielen: added new converter for TEI files provided by Jan Thielen.new_books/convert/tei_converter_generic.py,new_books/convert/helper/html2md.py: Add the possibility to pass options to themarkdownifyfunctionv.0.1.0:openiti.helper.yml: add support for empty lines and bullet lists in multiline valuesopeniti.new_books.convert.shamela_converter: fix bugs in shamela converterv.0.0.9.post1 (patch):openiti.helper.ara: fix bug in regex compilationv.0.0.9:openiti.new_books.convert: check and update all convertersopeniti.helper.ara: make counting characters in editorial sections optional (default: include Arabic characters in editorial sections)openiti.helper.yml: add custom error messages for broken and empty yml filesopeniti.git.git_util: add git utilities class, withcommitmethodv.0.0.8:openiti.new_books.add.add_books: fix import bugopeniti.new_books.convert: add converter for Noorlib html filesv.0.0.7:openiti.git.get_issues: change authentication from username/password to GitHub tokenopeniti.helper.ara: add function to normalize composite Arabic charactersopeniti.helper.uri: move functions for adding texts to the corpus to a new module,openiti.new_books.add.add_booksopeniti.helper.uri: fix bug in the character count function (did not work if execute==True)openiti.new_books.convert: restructured folder and moved helper functions into a new subfolder calledhelperopeniti.new_books.convert.generic_converter:reordered the mainconvert_filefunction and added inline documentationmadeconvert_files_in_folderfunction more flexibleopeniti.new_books.convert: added generic converters for shamela libraries, html and tei xml files, and custom converters for eShia and GRAR librariesopeniti.new_books.convert.shamela_converteropeniti.new_books.convert.html_converter_genericopeniti.new_books.convert.html_converter_eShiaopeniti.new_books.convert.tei_converter_genericopeniti.new_books.convert.tei_converter_GRARopeniti.new_books.convert.helper: added helper functions for the new converters:openiti.new_books.convert.helper.html2md_eShiaopeniti.new_books.convert.helper.html2md_GRARopeniti.new_books.convert.helper.tei2mdopeniti.new_books.convert.helper.bokv.0.0.6:openiti.helper.uri: use both Arabic character and token count in yml filesopeniti.helper.uri: add support for paths to files that are not in 25-years repos (e.g., for release)openiti.helper.uri: fix bugsadded Sphinx documentationv.0.0.5:openiti.helper.funcs: added Arabic token count functionopeniti.helper.uri: use Arabic token count instead of Arabic character count for yml file revision. Also, revise token count for every version yml file instead of only for version yml files that do not contain a count.v.0.0.4:openiti.helper.uri: removed the restriction on the use of digits in book titlesopeniti.helper.uri: added a check for empty yml filesopeniti.helper.yml: added documentation and doctestsopeniti.helper.yml: added check for empty yml files + changed splitting of yml files so that even unindented multi-line values can be correctly parsed.