Package: ekrhizoc
ekrhizocekrhizoc (E6c): A web crawlerContentsDefinitionUse CaseConfigurationDevelopmentTestingVersioningDeploymentProductionDefinitionεκρίζωση (Greek)
ekrízosi / uprooting, eradicationAlso known asE6c.Use CaseImplementation of a simple python web crawler.Input: URL (seed).Output: Simple textual sitemap (to show links between pages).RequirementsThe crawler is limited toonesubdomain (exclude external links).No use of web crawling libraries/frameworks (e.g. scrapy).(Optional) Use of HTML handling Libraries/Frameworks.Production-ready code.AssumptionsThe input URL (seed) is limited toonlyone at every run.The targeted URL(s) are static pages (no backend javascript parsing required).Links to be extracted from HTML anchor<a>elements.Valid links includeValid URLNon emptyMatches a valid url patternDoes not exceed theE6C_MAX_URL_LENGTHlength in charactersPossible to convert a relative urls to a full urlLink is not visited beforeLink is not part of an ignored file typeLink has the same domain as the seed urlLink is not restricted by the robots.txt fileDesignThis project implements a Basic Universal Crawler based on breadth first search graph traversal.ConfigurationBehaviour of the application can be configured via Environment Variables.Environment VariableDescriptionTypeDefault ValueE6C_LOG_LEVELLevel of logging - overrides verbose/quiet flagstring-E6C_LOG_DIRDirectory to save logsstring-E6C_BIN_DIRDirectory to save any output (bin)stringbinE6C_IGNORE_FILETYPESFile types of websites to ignore (e.g. ".filetype1,.filetype2")string".png,.pdf,.txt,.doc,.jpg,.gif"E6C_URL_REQUEST_TIMERTime (in seconds) to wait per request (not to populate server with multiple requests)float0.1E6C_MAX_URLSThe maximum number of urls to fetch/crawlinteger10000E6C_MAX_URL_LENGTHThe maximum length (character count) of a url to fetch/crawlinteger300DevelopmentConfigure your local developmentClonerepoon your local machineInstallcondaorminicondaCreate your local project environment (based onconda,poetry,pre-commit):$ make env(Optional) Update existing local project environment:$ make env-updateRun locallyOn a terminal, run the following (execute on project's root directory):Activate project environment:$ . ./scripts/helpers/environment.shRun the CLI usingpoetry:$ ekrhizocContribute[ Not Available ]Testing(part of CI/CD)[ Work in progress... ]To run the tests, open a terminal and run the following (execute on project's root directory):Activate project environment:$ . ./scripts/helpers/environment.shTo run pytest:$ make testTo check test coverage:$ make test-coverageVersioningIncrement the version number:$ poetry version {bump rule}where valid bump rules are:patchminormajorprepatchpreminorpremajorprereleaseChangelogUseCHANGELOG.mdto track the evolution of this package.The[UNRELEASED]tag at the top of the file should always be there to log the work until a release occurs.Work should be logged under one of the following subtitles:AddedChangedFixedRemovedOn a release, a version of the following format should be added to all the current unreleased changes in the file.## [major.minor.patch] - YYYY-MM-DDDeploymentPip packageOn a terminal, run the following (execute on project's root directory):Activate project environment:$ . ./scripts/helpers/environment.shTo build pip package:$ make build-packageTo publish pip package (requires credentials to PyPi):$ make publish-packageDocker imageOn a terminal, run the following (execute on project's root directory):Activate project environment:$ . ./scripts/helpers/environment.shTo build docker image:$ make build-dockerProductionFor production, a Docker image is used.
This image is published publicly ondocker hub.First pull image from docker hub:$ docker pull nichelia/ekrhizoc:{version}Execute CLI via docker run:$ docker run --rm -it -v ~/ekrhizoc_bin:/tmp/bin nichelia/ekrhizoc:{version} {command}This command mounts the application's bin (outcome) to user's root directory under ekrhizoc_bin folder.where version is the published application version