Package: arche
Archepip install archeArche (pronouncedArkey) helps to verify scraped data using set of defined rules, for example:Validation withJSON schemaCoverage (items, fields, categorical data, including booleans and enums)DuplicatesGarbage symbolsComparison of two jobsWe use it in Scrapinghub, among the other tools, to ensure quality of scraped dataInstallationArche requiresJupyterenvironment, supporting bothJupyterLabandNotebookUIFor JupyterLab, you will need to properly installplotly extensionsThen justpip install archeWhyTo check the quality of scraped data continuously. For example, if you scraped a website, a typical approach would be to validate the data with Arche. You can also create a schema and then set upSpidermonDeveloper Setuppipenv install --dev
pipenv shell
toxContributionAny contributions are welcome! Seehttps://github.com/scrapinghub/arche/issuesif you want to take on something or suggest an improvement/report a bug.ChangesMost recent releases are shown at the top. Each release shows:Added: New classes, methods, functions, etcChanged: Additional parameters, changes to inputs or outputs, etcFixed: Bug fixes that don't change documented behaviourNote that the top-most release is changes in the unreleased master branch on Github. Parentheses after an item show the name or github id of the contributor of that change.Keep a Changelog,Semantic Versioning.[0.3.6] (2019-07-12)AddedCategoriesrule with a plot showing unique values and count per field. By default,report_all()only includes fields which have less or equal to 10 unique values. Seehttps://arche.readthedocs.io/en/latest/nbs/Rules.html#Category-fields, #100Category documentationChangedArche.report_all()does not shorten report by default, addedshortparameter.Data is consistent with Dash and Spidermon:_type, _keyfields are dropped from dataframe, raw data, basic schema, #104, #106df.indexnow stores_keyinsteadbasic_json_schema()works withdeletedjobsstartis supported for Collections, #112enumis counted as acategorytag, #18Garbage Symbolssearches in str representation of nested fields instead of expanded df, #130Show real coverage difference (negative\positive) instead of absolute, #114FixedArche.glance(), #88Item links in Schema validation errors, #89Empty NAN bars on category graphs, #93data_quality_report(), #95Wrong number of Collection Items if it contains item 0, #112RemovedResponses Per Item RatioruleDeprecatedexpandparameter and removedflat_df, sinceGarbage Ruledeal with nested data itself, #133[0.3.5] (2019-05-14)AddedArche()supports any iterables with item dicts, fixing jsonschema consistency, #83Items.from_arrayto read raw data from iterables, #83ChangedIf reading from pandas df directly, store raw data in numpy array. See gotchashttp://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html#support-for-integer-naFixedRemoved[0.3.4] (2019-05-06)Fixedbasic_json_schema() fails with long1.0types, #80[0.3.3] (2019-05-03)AddedAccept dataframes as source or target, #69Changeddata_quality_report plots the same "Fields Coverage" instead of green "Scraped Fields Coverage"Plot theme changed from ggplot2 to seaborn, #62Same target and source raise an error, was a warning beforePassed rules marked with green PASSED.FixedOnline documentation now renders graphshttps://arche.readthedocs.io/en/latest/, #41Error colours are back inreport_all().RemovedDeprecatedArche.basic_json_schema(), usebasic_json_schema()Removed Quickstart.md as redundant - documentation lives in notebooks[0.3.2] (2019-04-18)AddedAllow reading private raw schemas directly from bitbucket, #58ChangedProgress widgets are removed before printing graphsNew plotly v4 APIFixedFailingCompare Prices For Same Urlswhen url isnan, #67Empty graphs in Jupyter Notebook, #63RemovedScraped Items History graphs[0.3.1] (2019-04-12)FixedEmpty graphs due to lack of plotlyjs, #61[0.3.0] (2019-04-12)FixedBig notebook size, replaced cufflinks with plotly and ipython, #39ChangedFields Coveragenow is printed as a bar plot, #9Fields Countsrenamed toCoverage Differenceand results in 2 bar plots, #9, #51:Coverage from job stats fields countswhich reflects coverage for each field for both jobsCoverage difference more than 5%which prints >5% difference between the coverages (was ratio difference before)Compare Scraped Categoriesrenamed toCategory Coverage Differenceand results in 2 bar plots for each category, #52:Coverage forfieldwhich reflects value counts (categories) coverage for the field for both jobsCoverage difference more than 10% forfieldwhich shows >10% differences between the category coveragesBoolean FieldsplotsCoverage for boolean fieldsgraph which reflects normalized value counts for boolean fields for both jobs, #53RemovedcufflinksdependencyDeprecatedcategory_fieldtag[2019.03.25]AddedCHANGES.mdnewarche.rules.duplicates.find_by()to find duplicates by chosen columnsimport arche
from arche.readers.items import JobItems
df = JobItems(0, "235801/1/15").df
arche.rules.duplicates.find_by(df, ["title", "category"]).show()basic_json_schema().json()prints a schema in JSON formatResult.show()to print a rule result, e.g.from arche.rules.garbage_symbols import garbage_symbols
from arche.readers.items import JobItems
items = JobItems(0, "235801/1/15")
garbage_symbols(items).show()notebooks to documentationChangedTags rule returns unused tags, #2basic_json_schema()prints a schema as a python dictDeprecatedArche().basic_json_schema()deprecated in favor ofarche.basic_json_schema()RemovedFixedArche().basic_json_schema()not usingitems_numbersargument2019.03.18Last release without CHANGES updates