Package: pycognaize
Cognaize SDKWelcome to Cognaize SDK. Cognaize SDK provides tools and
functionalities for creating, evaluating and deploying models into Cognaize platform.WebsiteDocumentationCognaize SDKprovides:Working withCognaize snapshotsWorking withOCR dataWorking withPDF filesWorking withimagesInstallationCognaize SDK can be installed using pip:pip install pycognaizeHave a look at thequick tutorialfor understanding main concepts of the SDK.Changelog[1.4][1.4.39] - 2024-02-27Handle tables inDocument.get_layout_text[1.4.38] - 2024-02-25AddDocument.get_layout_textmethod[1.4.37] - 2024-02-23AddGeniefor easy model testing[1.4.36] - 2024-02-21Handle pandas warnings and 1.2.0 compatibility issues.Handle M series Mac incompatibility issues.Warn for missing page image/ocr information.Addpyarrowas a requirement[1.4.35] - 2024-02-21Optimizepycognaize.common.utils.img_to_black_and_whiteTest coverage 85%[1.4.34] - 2024-01-23Fix create_lines function in Page classUpdate docs/requirements.txtUpdate python_versions.svg[1.4.33] - 2024-01-16Remove support for python 3.8 and lower. Add support for 3.10, 3.11 and 3.12Usemapinstead ofapplymapinDataFrameobjects[1.4.32] - 2024-01-10Fix load_page_ocr and load_page_images[1.4.31] - 2023-12-04Addclassesproperty to fields[1.4.30] - 2023-11-30Removecloudstorageiofrom the SDK[1.4.29] - 2023-11-06Changes os.path.join for s3 paths[1.4.28] - 2023-10-25Add support for scale output[1.4.27] - 2023-09-27Add support for s3 path resolution in Windows[1.4.26] - 2023-09-25kAddexclude_htmlto snapshot downloadAddincludeoption to snapshot downloader[1.4.25] - 2023-09-13Add unit test forspan_field.py[1.4.24] - 2023-09-13Create snapshot downloader class[1.4.23] - 2023-08-17Add tests for exclude optionsAdd tests for cloudstorageio hooks usageUpdate to cloudstorageio version 1.2.14[1.4.22] - 2023-08-16Add new method for document fetching[1.4.21rc1] - 2023-08-08Use new method from cloudstorageio for big snapshot download[1.4.20] - 2023-07-30Add replacing text-span datatype with text_span infrom_dictinput_fields of Document object[1.4.19] - 2023-07-30Update_to_dictandconstruct_from_rawfunctions inSpanFieldclass to set value when field does not have tag[1.4.18] - 2023-07-28Update_line_valuesattribute inSpanFieldclass in order to return sentences per lines[1.4.17] - 2023-07-19Update__add__function inExtractionTagclass in order to include the text between the twoBoxTagobjects selected[1.4.16] - 2023-07-14Addscaleproperty toNumericFieldclass[1.4.15] - 2023-07-12NumericField.valuewill return the calculated value if no field or tag values are available[1.4.14] - 2023-07-11Add exclude options for Snapshot download[1.4.13] - 2023-07-10Add calculated values toNumericField[1.4.12] - 2023-06-30Add missing function toCloudService[1.4.11] - 2023-06-28Ignore empty table tags inlangchain_loader.py[1.4.10] - 2023-06-28Separate LangchainLoader text blocks with new lines[1.4.10] - 2023-07-13Addto_text()functionality toDocumentclass that utilizes inputs fromPageLayoutmodel to create the text version of the document[1.4.9] - 2023-06-27Addto_string()functionality totable_tagsAdd langchain_loader util to convert PycognaizeDocumentobjects to LangchainDocumentobjects[1.4.8] - 2023-06-24Add field value and tag value to numeric field[1.4.7] - 2023-06-15Addocr_tags()andline_tags()toPage[1.4.6] - 2023-06-27Add replace_nans_with_empty_html_tags functionality in HTMLTableTag build df[1.4.5] - 2023-06-12Add re-login when AWS token is expired[1.4.4] - 2023-05-31Addanytreeto setup-requires insetup.cfg[1.4.3] - 2023-05-31Fix handling of section field with no tags[1.4.2] - 2023-05-30Fix handling of section field value when field does not have tags[1.4.1] - 2023-05-29Add section field and section tag functionality[1.4.0] - 2023-05-25Add classification labels functionality toFieldobjects[1.3][1.3.14] - 2023-05-10Add LinkField objectAdd returning group name in to_dict functionality of Field object[1.3.13] - 2023-05-01Add exclude folders option forlisdirinhtml_infoUpdate version cloudstorageio >= 1.2.8[1.3.12] - 2023-05-01Rename HTMLTag to HTMLTagABCRename TDTag to HTMLTagHandle out of table tags in XBRL[1.3.11] - 2023-04-28Add interface to create directory summary hashesAdd automatic snapshot hash creation forsnapshot.download[1.3.10] - 2023-03-31Add login command and code for submit to model registry[1.3.9] - 2023-03-30Refactor HTML._validate_path() to include try-except block[1.3.8] - 2023-03-28Improve html file path validation by adding a new check
in HTML._validate_path()[1.3.7] - 2023-03-24Match (xbrl) using xpath and indices in matches function ofmodel.py[1.3.6] - 2023-03-19Add functionality to run Model().execute_eval given
the ground truth document id[1.3.5] - 2023-03-11Make Confidence key lowercase.[1.3.4] - 2023-03-09Rename classConfidence key to Confidence in enums.[1.3.3] - 2023-03-07Check that the sum of confidence values is close to 1 instead of exactly 1.[1.3.2] - 2023-02-21Add is_xbrl attribute in document, to identify if document is XBRL or notModify assign indices functionality to handle XBRL tablesModify Model.matches() to also match with HTMLTagsAdd HTML._validate_path() to get valid path ofsource.htmlfileFix reading html file from S3Add tag_id attribute to HTMLTag[1.3.1] - 2023-02-14Update install requirements insetup.cfgRename environment variableCOGNAIZE_HOSTtoAPI_HOSTRaise error when trying to log in withoutAPI_HOSTenvironment variableAdd documentation and make cosmetic changes inlogin.py[1.3.0] - 2023-02-10Add XBRL supportAddspacytomodel-requirements.txtAddbs4torequirements.txt[1.2][1.2.9] - 2023-01-17Add class confidence functionality to tag objects[1.2.8] - 2023-01-09Update Numeric parser to handle float numbers with three or more
decimal numbersAdd "-" character handler in numeric parserChange all occurrences of 'cognaize' to 'Cognaize'[1.2.7] - 2022-12-22Change PyMuPDF version to support M1[1.2.6] - 2022-12-06Update numeric parser to better handle decimal numbers[1.2.5] - 2022-11-28Read page image height/width from document.jsonField raw value bug fix[1.2.4] - 2022-11-28Field object raw_value bug fix[1.2.3] - 2022-11-16Field object raw_value bug fix[1.2.3] - 2022-11-16Field object raw_value contains fields value[1.2.2] - 2022-11-11Add HTTP request timeout for genie model run[1.2.1] - 2022-11-08Add functionality for grouping fields by key[1.2.0] - 2022-10-11Modify assign indices functionality to correctly index tables located side by side[1.1][1.1.4] - 2022-10-06Add snapshot download to specified directory
functionalityAdd Page Section tag and field functionalities[1.1.3] - 2022-10-05Add login functionality to pycognaize[1.1.2] - 2022-09-12group_by_field returns list of fields with group_keyAdd minor improvement for NumericParserTied fields now only return unique fields[1.1.1] - 2022-09-05Fix field grouping with non-existing keyAdd field grouping with givenFieldobject[1.1.0] - 2022-08-25Added multiprocessing download of images and
ocr data[1.0][1.0.3] - 2022-08-23Change get_tied_fields, get_tied_tags,
get_first_tied_field, get_first_tied_tag methods to return also python namesFix get_first_tied_field_value and get_first_tied_tag_value methods to work
properly after changes[1.0.2] - 2022-08-23Fix get_tied_tags method[1.0.1] - 2022-08-22Fix get_tied_fields method[1.0.0] - 2022-08-15Update signature of base Field classUpdate constructor of SpanField class[0.3][0.3.66] - 2022-08-15Add tied_field and tied_tag functionality to document[0.3.65] - 2022-08-10Enhance numeric parser to handle strings like 0.01, add unittest[0.3.64] - 2022-08-09Add span field and span tag[0.3.63] - 2022-08-04Add grouping functionality for input and
output fields of document[0.3.62] - 2022-07-20Update GitHub workflow to publish documentation on release[0.3.61] - 2022-07-13Read image width and height from document.json instead of loading actual images for that[0.3.60] - 2022-07-03Fix seaborn issue[0.3.59] - 2022-07-03Remove opencv and seaborn from setup.cfg requirements[0.3.58] - 2022-06-20Deprecation decorator chooses version automatically[0.3.57] - 2022-06-20Fix _post_response_eval() method of Model[0.3.56] - 2022-06-20Remove OpenCV from main requirements (now in model-requirements)Add deprecation and module not found warning decoratorsGet rid of opencv dependenciesUpdate GitHub workflow to use model-requirements[0.3.55] - 2022-06-18Set logging level to debug for missing OCR or image files[0.3.54] - 2022-06-18Added for running workflows[0.3.53] - 2022-06-18Exclude tests from the package distroInclude white pixel in the distroRemove MANIFEST.inAdd virtualenv to dev-requirements.txtPublish in main pypi repository when running setup.sh[0.3.52] - 2022-06-15Integrate evaluation driver[0.3.51] - 2022-06-11Fix get_matching_table_cells_for_tag[0.3.50] - 2022-06-10Update documentation (logo, badges, etc.))[0.3.49] - 2022-06-06Add names to GitHub actions[0.3.48] - 2022-06-06Remove redundant information from README.md[0.3.47] - 2022-06-06Update setup.sh to create wheel and upload to pypiSetup.sh performs doctests as part of the build process[0.3.46] - 2022-06-03Added tutorial about working with tags, and PDFMade updates to documentation[0.3.45] - 2022-06-03Obfuscate data in pycognaize testsUpdate names in GitHub actions[0.3.44] - 2022-05-31Create tutorial about leveraging tables in cognaize SDKAdd logo and favicon to documentationAdd supported python version badge to readmeAdd logo to readme[0.3.43] - 2022-05-28Remove outdated modules (ocr.py and recipe.py)Remove Dockerfile and outdated build scriptsAdd dev-requirements.txt, update badges[0.3.42] - 2022-05-27Add badge generating script to show in ReadmeUpdate Readme to show badge, documentation and tutorialsAdd sphinx doctests to GitHub actions[0.3.41] - 2022-05-25Updated homepage of the documentationAdded versioning to RTD. (NOT FINAL. CHECKS VERSIONS FROM GIT WHICH…)Added blank tutorial pagesUpdated sidebar toc tree structureAdded doctests to quick tutorialdded sphinx.ext.doctestUpdated create_docs to include new_autosummarydirectory and create doctest[0.3.40] - 2022-05-20Fix styling and refactor in order to pass flake8 checks[0.3.39] - 2022-05-20Add backquotes in changelogGroup changelog entriesChangelog ordered from the latest version to firstAdd script to deploy the docsAddquick_tutorial.rstfileUpdate index.rst to include links to general sectionsAdd myst-parser to docsrequirements.txtChange Markdown parser to myst-parserAddquickstart.rstAdd reading.mdfiles for changelogSeparateinstallation.rst[0.3.38] - 2022-05-19Configure sphinx for generating proper API referenceCreate general docs outline, add generatedautosummaryrst files in ignore files[0.3.37] - 2022-05-16Added docstrings and type hints to modules[0.3.36] - 2022-05-16Rename the package to pycognaize[0.3.35] - 2022-05-13Added GitHub workflows for linting and testingChanged docstrings in snapshot.py and lazy_dict.py[0.3.34] - 2022-05-12Fix all tests[0.3.33] - 2022-05-11Fix requirements and setup.sh[0.3.32] - 2022-04-11Add opencv-python-headless==4.0.1.23 in requirements for avoiding ImportError: cannot import name
'_registerMatType' (only for usage in table_detection)[0.3.31] - 2022-04-11Bring back using opencv in bytes_to_array, string_to_array, img_to_black_and_white for
stick_word_boxes functionality (only for usage in table_detection)[0.3.30] - 2022-04-11Change TableTag to take cell data, not use table dividersAdd tests corresponding to changed TableTag[0.3.29.a0] - 2022-04-06Fix in get_table_title to give not first 8 rows of page while 8 rows above table[0.3.29] - 2022-04-06Add numeric parser in common[0.3.28] - 2022-03-04Remove opencv from requirements, add some utils[0.3.27] - 2022-02-10Set requirement pymupdf<=1.19.4 in setup.cfg[0.3.26] - 2022-02-04Fix pymupdf conflicting version issue(1.19.4)[0.3.25] - 2021-11-13Add raw_value to TextField[0.3.24] - 2021-11-13call build_df inTableTag.__init__to skip corrupted table tags[0.3.23] - 2021-11-13call TableTag._build_df in init so that corrupt table tags are not created on snapshot read[0.3.22] - 2021-11-13ImplementTableTag.__getitem__Define TableTag.raw_dfCache property TableTag.df on accessDo not build_df inTableTag.__init__[0.3.21] - 2021-11-12Fix Document.get_table_cell_overlap (look both in input and output fields, fix the iou page check issue)[0.3.20] - 2021-11-10Add Document.metadata attribute[0.3.19] - 2021-11-10Remove assertion tests for fields and tags where applicableSet all warnings to debug level in tag.pyDo not fail on invalid tag json data, but skip the tag (all logs are on debug level)[0.3.18] - 2021-11-02Enforce utf-8 encoding when reading document.json locally[0.3.17] - 2021-10-29Add Document.to_pdf() featureAdd annotate_pdf() function in document.pyAdd Unittests for Document.to_pdf()Fix requirement (fitz to pymupdf)[0.3.16] - 2021-10-22Make evaluate method of Model class abstract[0.3.15] - 2021-10-21Add handling repeating field and group cases in evaluate functionality of model[0.3.14] - 2021-10-20Rename Model.predict_based_on abstract method toModel.copymethodRemove predict_based_on method from test_model ExampleModel class[0.3.13] - 2021-10-12Add Model.execute_based_on_match, Model.predict_based_on, and Model._post_response methodsAdd separateIndex._storemethodAddresponse_to_dictmethod to index class for transformation of the response to needed format - {doc_id: encoding}Updateexecute_based_on_matchmethod in model class to get document object for matched base document"Remove INDEX from ModelChange Index to create fields for matched document ID and confidence[0.3.12] - 2021-10-11Remove ssl verification from GET/POST requests[0.3.11] - 2021-10-04Add tests for pycognaize.common.utils.intersects[0.3.10] - 2021-10-03Fix pycognaize.common.utils.intersects[0.3.9a24] - 2021-09-06Fixed the issue of local running the tests[0.3.9a23] - 2021-09-01LаzyDict.__getitem__returns None, if reading the document failsDefine the return value type[0.3.9a22] - 2021-08-31Add Index class for document-index abstractionAdd unittests for Index (93% coverage)[0.3.9a21] - 2021-08-31add return_tags functionality in get_ocr_formatted, _create_lines,
search_text, extract_area_words of Pageadd image_bytes propertyremove assigning results of get_image() ang get_ocr() to hidden attributes[0.3.9a20] - 2021-08-26Added test for execute_genie_v2[0.3.9a19] - 2021-08-24sortedself._idsfor lazy_dict.py in line 18[0.3.9a18] - 2021-08-24Added test_lazy_dict.py[0.3.9a17] - 2021-08-24Added tests for document.ocr.py.[0.3.9a16] - 2021-08-20Added missing tests for table_tag.[0.3.9a15] - 2021-08-20Added tests for draw. Corrected an issue in page.py.[0.3.9a14] - 2021-08-20Corrected the issue in test_utils.py. Changed the writen code that relies on the ordering of os.listdir.[0.3.9a13] - 2021-08-20Add tests for Page (90% coverage)[0.3.9a12] - 2021-08-20Correct tag Euclidean distance method by changing the private variables into public[0.3.9a11] - 2021-08-18Added missing test methods for Tag, utils, Cell, AreaField, Document, and Field (89% coverage).[0.3.9a10] - 2021-08-16Fix tag Euclidean distance method[0.3.9a9] - 2021-08-13Added area argument to page.search_text() to specify scope[0.3.9a8] - 2021-08-13Add image_arr, image_height, image_width, ocr_raw properties[0.3.9a7] - 2021-08-11Added getter and setter for field group_key[0.3.9a6] - 2021-08-10Add an option for image size in page.draw() and set a larger size as defaultAdd OS specific behavior for preview_imgRemove unnecessary exceptions[0.3.9a5] - 2021-08-10Add tag euclidean distance method[0.3.9a4] - 2021-08-09Add evaluation and unittests including content only metricsAdd ConfusionMatrix and heatmap drawing function[0.3.9a3] - 2021-08-08Add EnvConfigEnum.SNAPSHOT_IDget snapshot_path using snapshot_idrefactor LOCAL_SNAPSHOT_PATH to SNAPSHOT_PATHUpdate snapshot.py tests[0.3.9a2] - 2021-08-08page.search_text() did not find certain substrings present in page.free_form_text(). Found two reasons for this behavior.The list of ocr-data passed tofind_frirst_word_coordwas page.ocr('words'), which has the entries sorted by word_id. This makes the sort flag of the function obsolete, and second it leads to cases in which the coordinates of sub-strings from page.free_form_text() cannot be found using the function the order of the words in page.free_form_text() is unrelated to word_id.
For example a sub-string of page.free_form_text() might be “brown fox” with the word_id of “brown” being equal to 3 and “fox” being equal to 8. In this case find_first_word_coords would not find the coordinates, as it would break the for-loop as the word with word_id 4 is not “fox”.
This behavior was fixed by passing the ocr data in the same order as in page.free_form_text(), still giving the option to sort it by word_id using the sort-flag.Inside the find_first_word_coord function the words of the sub-string were always put through a cleanup regex before being compared to the ocr_text (which was not cleaned up if the clean-flag was set to false). This leads to cases in which a sub-string such as “Phone: 12345” would not be found as “Phone:” would be cleaned up to “Phone”.
This was fixed by either putting the words of the sub-string as well as the values for ocr_text through a cleanup regex or neither of them, depending on the clean-flag.[0.3.9a1] - 2021-08-06Implement and/or update all tests for 0.3.7 versions (ALL TESTS PASS)Optimize imports[0.3.8a5] - 2021-08-03 (0.3.8 versions do not include the changes in 0.3.7 versions)Fix page.search_text()[0.3.8a4] - 2021-07-18 (0.3.8 versions do not include the changes in 0.3.7 versions)AddModel.evaluateFix changelog wrong years (Incorrect 2020 years changed to 2021)[0.3.8a3] - 2021-06-24 (0.3.8 versions do not include the changes in 0.3.7 versions)Fix table divider offsets and interruption coordinates (FIXES THE BUG FROM 0.3.8a2)[0.3.8a2] - 2021-06-24 (0.3.8 versions do not include the changes in 0.3.7 versions)Fix table divider offsets and interruption coordinates (BUGGED VERSION)[0.3.8a1] - 2021-05-23 (0.3.8 versions do not include the changes in 0.3.7 versions)FixTag.intersectsmethod[0.3.7a9] - 2021-07-07AreaField will raise a warning if the input value field is not a string and set it to empty string (if the field has no tags)[0.3.7a8] - 2021-05-30FixAreaField.value[0.3.7a7] - 2021-05-26Fix get item in lazy_dict (required path)[0.3.7a6] - 2021-05-23FixTag.intersectsmethod[0.3.7a5] - 2021-05-13Fix NaN issue in execute_genie_v2 post request json[0.3.7a4] - 2021-05-10Fix table cell value population issue[0.3.7a3] - 2021-05-09If page image or ocr files cannot be found, use an empty ocr/ 1 white pixel image instead[0.3.7a2] - 2021-05-08Add execute_genie_v2 for executing genie with airflow[0.3.7a1] - 2021-05-08Page object uses absolute path and allows lazy-loading from cloud (all tests pass)Adjust all filename conventions to work with original image/ocr storage name conventions[0.3.6] - 2021-03-29Adjust tests to count for the Range margin (all tests pass)[0.3.6a4] - 2021-03-05Modify the margin in Range.to_dict, set margin to 0.15[0.3.6a3] - 2021-03-05Add margin to table cells in TableTag._build_cellFix ordering information in digesterFix typing annotation for OrderedDict outputChange document.x and document.y into OrderedDict, use global ordering in digester[0.3.6a2] - 2021-02-24Add tests for Tag/ExtractionTag (coverage 87%)[0.3.6a1] - 2021-02-24Fox group_key typing annotation in Field objects[0.3.5] - 2021-02-23Store pdf in the snapshot[0.3.4] - 2021-02-22Add group_key optional argument to Field objects[0.3.3] - 2021-02-16All tests pass (84% coverage)Snapshot creator with threadingUse mongomock for DB tests[0.3.2] - 2021-02-09Update tests for storage[0.3.1a4] - 2021-02-03Add threading toSnaphotBuilder[0.3.1a3] - 2021-02-02Fixtest_digestor.pyto work with the correct document bson fileAll tests pass (82% coverage)[0.3.1a2] - 2021-02-02Set DB.find call arguments no_cursor_timeout=True, batch_size=10 in SnapshotBuilder in order to avoid CursorNotFound timeout errors[0.3.1a1] - 2021-01-29Add lines, search_text, extract_area_words methods to PageAdd unittests for lines, search_text and extract_area_words methodsAdd infer_rows_from_words, clean_ocr_data, find_first_word_coords, intersects, compute_intersection_area methods in utils.py[0.3.1a0] - 2021-01-28Add unittest for Document.from_dictOptimize digester output_fields lookupAdd stick_coords option to Page.get_ocr_formattedAdd opencv requirement[0.3.0b20] - 2021-01-16Assign document inexecute_geniewhen callingmodel.predict[0.3.0b19] - 2021-01-12Assigning recipe output fields in digester for better performance[0.3.0b18] - 2021-01-06Update digester to set id-s of the original fields from the blueprintChangedocument.tagimports to relative[0.3.0b17] - 2021-01-03Model.predict in Model.execute_genie uses positional arguments[0.3.0b16] - 2021-01-03Load LazyDocumentDicts as bsonMake sure page numbers are integers[0.3.0b15] - 2021-01-03Fix SnapshotBuilder.save_doc_json_to_snapshot document_id key[0.3.0b14] - 2021-01-03Update Document.to_dict document_id key in metadata[0.3.0b13] - 2021-01-03Add field name and ID in Field.to_dict implementations[0.3.0b12] - 2021-01-03Add pages argument to construct_from_raw[0.3.0b11] - 2021-01-03Fix Document.from_dict typo (construct_from_raw method call)[0.3.0b10] - 2021-01-03Define Field data_types in to_dict methodsAdd area to IqDataTypesEnum David A minute ago[0.3.0b9] - 2021-01-03Move FieldMapping tofield.__init__Fix circular import in fieldsUse super().to_dict() in Field objects[0.3.0b8] - 2021-01-03Fix Document.from_dict page iterationAdd field types to Field.to_dict methodsTableField allows no tags when calling to_dict method[0.3.0b7] - 2021-01-02Update Dockerfile entrypoint to pycognaize.app.restFix SnapshotBuilder.create_document_zip cls.DB assignment expression[0.3.0b6] - 2021-01-02Update changelog, fix all versions to 0.3.0b6[0.3.0b4] - 2021-01-02Update all tests (75% coverage)Merge branch 'master' into major_refactorAdd Snapshot topycognaize.__init__Use scandir in DocumentBuilder._populate_pagesUpdate import statement for MappingFix SnapshotBuilder to work with new Snapshot classRemove FieldMapping from DocumentBuilder, use a separate module insteadSnapshot uses lazy_dict for reading individual documentsUsetempfilemodule in model.pyAdd doc_file (document.json) to SnapStorageEnumAdd AreaField tofield.__init__Add to_dict and from_dict methods to Document classAdd test coverage in toxMerged in add_test_for_overwriting_snapshot_in_s3 (pull request #47)pull updates from major refactor and merge with current branch, remove test_service, add test_store_snapshot_with_same_nameMerge branch 'major_refactor' into add_test_for_overwriting_snapshot_in_s3add unittests to test snapshot overwriting, change overwriting log messageAllow 'from pycognaize import Model'Add doctests to text_field moduleUpdate sphynx conf.pyUpdate build_docs.sh to also build pdf documentation fileUpdate setup.sh logsMove all setup configurations from setup.py to setup.cfgbuild_docs.sh generated html and pdf documentationsAdd doc/source/generated/ to .*ignore files[0.3.0b3] - 2020-12-28Add script for building sphinx docsMinor docstring changes to TaghshiftandvshiftmethodsSnapshotBuilder.DB added only on function call, to speed up module importsAdd a single doctest to TextField constructorAdd simplejson to requirementsAdd Model.execute_genie method[0.3.0b1] - 2020-12-27Major refactored versionDocument > DocumentBuilder
(DocumentBuilder has no instance, only methods for creating Documents,
which are now equivalent to DocumentDataclass objects)DocumentDataclass > DocumentSnapshotProcessor > SnapshotBuilderChanged folder structure (no services package)DataSnapshot > Snapshot, DataRecipe > RecipeAdd tox configuration for py36, py37, py38, py39, pypyAdd 'MANIFEST.in' (required for 'TOX' to run properly)Add 'setup.cfg' for 'pytest''./setup.sh' builds and pushes a version, only if no tests failUpdate README.mdAll tests pass on py36, py37, py38, py39, pypy[0.2][0.2.5a4] - 2020-12-18Add docker push command in build.sh[0.2.5a3] - 2020-12-17Change rest service to threaded=False[0.2.5a2] - 2020-12-15Checkpoint version[0.2.5a1] - 2020-12-15Delete numpy from req-s[0.2.5a0] - 2020-12-04Update _build_df method in TableTag[0.2.4] - 2020-11-27Fix srcFieldId log in Document.get_fields_by_id to print field id instead of the whole field[0.2.3] - 2020-11-27Fix src_field_id issue indigestor.py[0.2.2.a7] - 2020-11-27All tests are fixed and running[0.2.2.a6] - 2020-11-18Fix issue in get_ocr_formatted[0.2.2.a5] - 2020-11-13Add setup.sh scriptUse cloudstorageio>=1.1.2 which supports uploading 5GB+ files to s3Do not store TableTag ocr (makes the pickle dumps way too big for documents with many tables)[0.2.2.a4] - 2020-11-07Fix typing annotation forDocumentDataclass._pagesAdd propertyAreaField.valueChangesuper().tagstoself.tags[0.2.2.a3] - 2020-11-01Do not storeTableTag.df, build it on callUpdate Range unittest (remove unnecessary error raising test cases)[0.2.2.a2] - 2020-11-01Add area, height, width to (Cell)Range objectsAdd support for comparing Tag and (Cell)Range objects in magic methods of TagIn create_document_zip, if the recipe retrieved from DB is empty, through a ConnectionError Add get_table_cell_overlap to DocumentDataclass[0.2.2.a1] - 2020-10-31Initiate database_setup on function call instead of import statement[0.2.2.a0] - 2020-10-14Adjust problematic OCR in Page.get_ocr_formatted (if left >= right, right = left + 1, same for top/bottom)[0.2.1.a3] - 2020-10-28-Optionally build df for TableTag using ocr data[0.2.1.a2] - 2020-10-22Add _build_df method in TableTag[0.2.1.a1] - 2020-10-08Catch validation errors for TableTag[0.2.1.a0] - 2020-09-23Adjust coordinates smaller than 0 and bigger than 100[0.2.0.a12] - 2020-09-14Always remove snapshot zip before creating a new one[0.2.0.a11] - 2020-09-06Update digester template check[0.2.0.a10] - 2020-08-20Use proper relative_path in populate_pagesAdd to_dict in AreaField[0.2.0.a9] - 2020-08-20Include data_recipe in DataSnapshot[0.2.0.a8] - 2020-08-20Add value argument in construct_from_raw method in DateField[0.2.0.a7] - 2020-07-30Filter fields with repeat_parent instead of source field id[0.2.0.a6] - 2020-07-28SnapshotProcessor raises error in create_document_zip if document not found in the DB[0.2.0.a5] - 2020-07-28Attribute value of TextField and DateField are strings[0.2.0.a4] - 2020-07-27Add to_dict method to DateField[0.2.0.a3] - 2020-07-24Fix construct_from_raw in ExtractionTagImplement to_dict for numeric_field[0.2.0.a2] - 2020-07-24Add generated ObjectId-s in to_dict methods, update readme[0.2.0.a1] - 2020-07-23Add additional include-services argument tosetup.py[0.2.0.a0] - 2020-07-23Add make_document_snap endpointImplement DbStoragedatabase_setup raises an error if failedRenamerest.apptoservice.restImplement to_dict method for TableField TableTag and cell Range objectsAdd template_ids property in DataRecipeAdd IqTableDividerEnum and update fields of other enums, including IqTableTagEnumChange cloudstorageio to version 1.0.10 in requirementsAdd digest_results functionAdd Storage abstract classAdd to_dict methods in ExtractionTag and TextFieldAdd IqFieldKeyEnum and add to_dictabstractmethodsin Tag and Field abstract classesUpdate readme to usenosetestscommandAdd abstract method decorators to methods in Tag, Field and Model abstract classesMake package exclusion dynamic in setup.py[0.1][0.1.9.alpha1] - 2020-07-16Fix get_ocr_formatted in Page classModel object's predict method takes document_dataclass as inputMajor cosmetic change, full PEP8 compliance, except max line lengthRemove spreading_document, build and tag_utils modules[0.1.9.alpha] - 2020-07-15Add Model abstract classUpdate README.md packaging instructions and setup.pyUpdate tests, comment DocumentDataclass tests, until proper setUp/tearDown is implementedUpdate SnapshotProcessor import in the rest.appRemove unnecessary methods from Document, remove outdated testsAdd AreaFieldAdjustments after renaming an IqTableEnum to IqCollectionEnumTableTag optionally uses raw cell data to construct cell ranges, change keys to 1-based indexChangereprandstrmethods in TableFieldAdd IqCellKeyEnum and IqTableFieldEnum[0.1.8.alpha] - 2020-07-15Update make_snap endpoint to work with SnapshotProcessor, add fetch_document_zip endpointAdd SnapshotProcessorAdd instruction for pushing to fury and pip installMinor cosmetic changes in ocr.pyRemove source_id from Document object constructorMake tag parameter optional in TableFieldRemove raw parameter from DataSnapshot constructorAdd collections to IqDatasetKeyEnumRemove CellTag[0.1.7.alpha] - 2020-07-14repr for Range includes the valueAdd methods for constructing tags and fields from raw dictionariesAdd OCRData and modify build_cell function using dividersImplement build rangesCell ranges addedRemove an outdated comment from DocumentDataclasAdd TableDivider objectAdd test for DocumentDataclassModify constructors for Field and Tag objectsAdd IqTagKeyEnum to enumsDocument.get_y returns a list of fields instead of a single fieldAdd get_ocr_formatted method in Pageadd src_field_id to IqDocumentKeysEnumMove database_setup into a separate module[0.1.6.alpha] - 2020-07-03Store relative path in page objects, remove SNAP_STORAGE_PATH constant and use env variable everywhere, fix some typos in TODOsAdd document_src, document_id and pages as properties in DocumentDataClassImprove DataSnapshot api, add document_dataclasses and get methodsImages and ocr folders are document ids instead of document srcPickled snapshot is saved assnap.pickleinstead ofsnap..pickleAdd StoredSnapshotExceptionAdd src attribute in document, put NoneField in a separate moduleFix Page repr[0.1.4.alpha] - 2020-07-02Improve log formatRedefine Snapshot, return traceback if rest.app failsStore images and ocr with snapshotChange pymongo to pymongo[srv] in requirements, fix DB name splitting iniq.__init__Remove db dependency from documentsRename Snapshot to DataSnapshot, Pipeline to DataPipeline, Recipe to DataRecipeAdd test for DocumentDataclassadd load_bson_by_path in utilsAdd document and recipebsonsto test resources[0.1.2.alpha] - 2020-06-10Use rest endpoint for get_data in DataSnapChange defaults for SNAP_STORAGE_PATH and DEFAULT_DB_URL, add back DEFAULT_DATASNAP_ENDPOINTFix TypeError message inDataPipe.updateAdd IQ_SNAP_STORAGE_PATH back to EnvConfigEnumAdd pydrive to requirements in order to solve issue with cloudstorageio, should be changed once cloudstorageio is updated[0.1.3.alpha] - 2020-06-26Add DocumentDataclassAdd get_x, get_y methods to DocumentRename DataRecipe to Recipe, add input_fields, output_fields attributes to RecipeRename snap to snapshotAdd raw_field_type in IqRecipeEnumCosmetic changes in DateFieldAdd NoneField[0.1.1] - 2020-06-08Modify rest.app to make snapshots through a shared volumeImplementDataSnap.createand DataSnap initialization through a shared volumeModify DataPipe in order to be inherited by DataSnapAdd cloudstorageio to requirementsAdd SnapshotPathMissingException and SnapshotExistsExceptionRemove IQ_DATASNAP_URL and add IQ_SNAP_STORAGE_PATH to EnvConfigEnumRemove igraph dependencies from DockerfileAdd DATASET_TYPE to DataSnap[0.1.0] - 2020-05-28Allow to import dataset types frompycognaize.datasetsAdd DataSnapRefactor the package structure[0.0][0.0.9] - 2020-05-26Add Dockerfile and build.sh for building docker image with hash[0.0.8] - 2020-05-24Fix DEFAULT_DB error handling, add rest API for snap[0.0.7] - 2020-05-23Add raw and construct_from_raw methodsAdd data snap serialization and deserialization[0.0.6] - 2020-05-22Fix parse_periods in SpreadingDocument[0.0.5] - 2020-05-22Change df property to return CellTag objects[0.0.4] - 2020-05-22Add get_document_periods methodAdd df property to TableTagInherit SpreadingDocument class from Document[0.0.3] - 2020-05-17Change__repr__in Field object, add TODO for document name column in DataRecipeDefine a DEFAULT_DB object, with a single env variable - IQ_DB_URL[0.0.2] - 2020-05-15Created DataRecipe[0.0.1]Created Tag, Field, Page and Document abstractions for pycognaize