Package: ek-scraper
ek-scraperSimple scraper for kleinanzeigen.de searches with notifications for new ads.InstallationInstall this package from PyPi in a separate virtual environment usingpipx.pipxinstallek-scraperUsageFor the full usage check theek-scraper --helpcommandCreate a configuration file usingek-scrapercreate-config<path/to/config.json>The example configuration file will look like this:{"filter":{"exclude_topads":true,"exclude_patterns":[]},"notifications":{"pushover":{"token":"<your-app-api-token>","user":"<your-user-api-token>","device":[]},"ntfy.sh":{"topic":"<your-private-topic>","priority":3},},"searches":[{"name":"Wohnungen in Hamburg Altona","url":"https://www.kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497","recursive":true}]}SeeConfigurationfor details on all configuration options.Configure one or more searches in thesearchessection of the configuration,
seeSearchesfor more detailsConfigure notifications in thenotificationssection of the configuration,
seeNotificationsfor details on notification configuration(Optional) Configure filters in thefiltersection of the configuration,
seeFilterfor more detailsRun the following command to initialize the data store without sending any notifications:ek-scraperrun--no-notificationspath/to/config.jsonAfterwards, runek-scraperrunpath/to/config.jsonto receive notifications according to yournotificationsconfiguration.DevelopmentFollow the steps below to set up a development environment for this project.Clone this repositorygitclonegit@github.com:jonasehrlich/ek-scraper.gitChange directory into the repositorycdek-scraperCreate a virtual environment usingpoetrypoetryinstall(Optional) Install pre-commit environment$pre-commit[INFO]Installingenvironmentforhttps://github.com/pre-commit/pre-commit-hooks.[INFO]Onceinstalledthisenvironmentwillbereused.[INFO]Thismaytakeafewminutes...[INFO]Installingenvironmentforhttps://github.com/psf/black.[INFO]Onceinstalledthisenvironmentwillbereused.[INFO]Thismaytakeafewminutes...
CheckYaml...........................................(nofilestocheck)Skipped
FixEndofFiles.....................................(nofilestocheck)Skipped
TrimTrailingWhitespace.............................(nofilestocheck)Skipped
black................................................(nofilestocheck)SkippedConfigurationSearchesSearches can be configured in thesearchesarray of the configuration file.
Each of the searches can be configured with the following parameters.NameDescriptionnameName of the search, use a descriptive one (required)urlURL of the first page of your search (required)recursiveWhether to follow all pages of the search result(optional, defaults to true)FilterFilters can be configured in thefiltersection of the configuration file to exclude specific ads
from your scrape results on the client side. The following settings can be configured.NameDescriptionexclude_topadsWhether to exclude top ads from the results (optional, defaults to true)exclude_patternsCase-insensitive regular expression patterns used to exclude ads (optional)NotificationsNotifications can be configured in thenotificationssection of the configuration file.Push notifications usingPushoverek-scrapersupports push notifications to your devices usingPushover.
For further information on the service check their terms and conditions.The implementation forPushovernotifications will send a single notification per search, if new
ads are discovered.To configurePushoverfor notifications from the scraper, first register at the service and create
an application (e.g.ek-scraper). To use the service inek-scraper, add thepushoverobject
to thenotificationsobject in your configuration file and fill the API tokens. Selection of the
devices which will receive the notifications, is supported using thedevicearray.NameDescriptiontokenAPI token of the Pushover app (required)userAPI token of the Pushover user (required)deviceList of device names to send the notifications to(optional, defaults to all devices)Push notifications usingntfy.shek-scrapersupports push notifications to your devices usingntfy.sh.
For further information on the service check their terms and conditions.The implementation forntfy.shnotifications will send a single notification per search, if new
ads are discovered.To configurentfy.shfor notifications from the scraper, define a topic and subscribe to it in the
mobile app.Note that topic names are public, so it's wise to choose something that cannot be guessed easily.
This can be done by including a UUID, e.g. by running the following command in your shell:echo"ek-scraper-$(uuidgen)"To use the service inek-scraper, add thentfy.shobject to thenotificationsobject in your
configuration file and add the topic you previously subscribed to.NameDescriptiontopicTopic to publish the notifications topriorityPriority to send the notifications with (optional, defaults to 3)Runningek-scraperregularlyIt should be avoided to run the tool too often to avoid getting your IP address blocked bykleinanzeigen.deIn order to runek-scraperregularly on a Unix-like system, configure it as a cronjob.To configure a cronjob, runcrontab-eEdit the crontab table to run the command you want to run. A handy tool to check schedule
configurations for cronjobs iscrontab.guru.For more information on configuring cronjobs use your favorite search engine.