Package: rentswatch-scraper
This package provides an easy and maintenable way to build a
Rentswatch scraper. Rentswatch is a cross-borders investigation that collects data on flat rents in Europe. Its scrapers mainly focus on classified ads.How to installInstall usingpip…pip install rentswatch-scraperHow to useLet’s take a look at a quick example of using Rentswatch Scraper to
build a simple model-backed scraper to collect data from a website.First, import the package components to build your scraper:#!/usr/bin/env pythonfromrentswatch_scraper.scraperimportScraperfromrentswatch_scraper.browserimportgeocode,convertfromrentswatch_scraper.fieldsimportRegexField,ComputedFieldfromrentswatch_scraperimportreportingTo factorize as much code as possible we created an abstract class that
every scraper will implement. For the sake of simplicity we’ll use adummy websiteas follow:classDummyScraper(Scraper):# Those are the basic meta-properties that define the scraper behaviorclassMeta:country='FR'site="dummy"baseUrl='http://dummy.io'listUrl=baseUrl+'/rent/city/paris/list.php'adBlockSelector='.ad-page-link'Without any further configuration, this scraper will start to collect
ads from the list page ofdummy.io. To find links to the ads, it
will use the CSS selector.ad-page-linkto get<a>markups and
follow theirhrefattributes.We have now to teach the scraper how to extract key figures from the ad
page.classDummyScraper(Scraper):# HEADS UP: Meta declarations are hidden here# ...# ...# Extract data using a CSS Selector.realtorName=RegexField('.realtor-title')# Extract data using a CSS Selector and a Regex.serviceCharge=RegexField('.description-list','charges : (.*)\s€')# Extract data using a CSS Selector and a Regex.# This will throw a custom exception if the field is missing.livingSpace=RegexField('.description-list','surface :(\d*)',required=True,exception=reporting.SpaceMissingError)# Extract the value directly, without using a RegextotalRent=RegexField('.description-price',required=True,exception=reporting.RentMissingError)# Store this value as a private property (begining with a underscore).# It won't be saved in the database but it can be helpful as you we'll see._address=RegexField('.description-address')Every attribute will be saved as an Ad’s property, according to the Ad
model.Some properties may not be extractable from the HTML. You may need to
use a custom function that received existing properties. For this reason
we created a second field type namedComputedField. Since the
properties order of declaration is recorded, we can use previously
declared (and extracted) values to compute new ones.classDummyScraper(Scraper):# ...# ...# Use existing properties `totalRent` and `livingSpace` as they were# extracted before this one.pricePerSqm=ComputedField(fn=lambdas,values:values["totalRent"]/values["livingSpace"])# This full exemple uses private properties to find latitude and longitude.# To do so we use a buid-in function named `convert` that transforms an# address into a dictionary of coordinates._latLng=ComputedField(fn=lambdas,values:geocode(values['_address'],'FRA'))# Gets a the dictionary field we want.latitude=ComputedField(fn=lambdas,values:values['_latLng']['lat'])longitude=ComputedField(fn=lambdas,values:values['_latLng']['lng'])All you need to do now is to create an instance of your class and run
the scraper.# When you script is executed directlyif__name__=="__main__":dummyScraper=DummyScraper()dummyScraper.run()API DocclassAdAttributesAs seen above, every Ad attribute might be used as a Scraper attribute to declare which attribute extract.NameTypeDescriptionstatusString“listed” if needs more scraping, “scraped” if it’s donesiteStringName of the websitecreatedAtDateTimeDate the ad was first scrapedsiteIdStringThe unique ID from the site where it’s scrapped fromserviceChargeFloatExtra costs (heating mostly)baseRentFloatBase costs (without heating)totalRentFloatTotal costlivingSpaceFloatSurface in square meterspricePerSqmFloatPrice per square meterfurnishedBoolTrue if the flat or house is furnishedrealtorBoolTrue if realtor, n if rented by a physical personrealtorNameUnicodeThe name of the realtor or person offering the flatlatitudeFloatLatitudelongitudeFloatLongitudebalconyBoolTrue if there is a balcony/terrasseyearConstructedStringThe year the building was builtcellarBoolTrue if the flat comes with a cellarparkingBoolTrue if the flat comes with a parking or a garagehouseNumberStringHouse Number in the streetstreetStringStreet name (incl. “street”)zipCodeStringZIP codecityUnicodeCityliftBoolTrue if a lift is presenttypeOfFlatStringType of flat (no typology)noRoomsStringNumber of roomsfloorStringFloor the flat is atgardenBoolTrue if there is a gardenbarrierFreeBoolTrue if the flat is wheelchair accessiblecountryStringCountry, 2 letter codesourceUrlStringURL of the pageclassScraperMethodsThe Scraper class defines a lot of method that we encourage you to
redefine in order to have the full control of your scraper behavior.NameDescriptionextract_adExtract ads list from a page’s soup.failPrint out an error message.fetch_adFetch a single ad page from the target website then create Ad instances by callingèxtract_ad.fetch_seriesFetch a single list page from the target website then fetch an ad by callingfetch_ad.find_ad_blocksExtract ad block from a page list. Called withinfetch_series.get_ad_hrefExtract a href attribute from an ad block. Called withinfetch_series.get_ad_idExtract a siteId from an ad block. Called withinfetch_series.get_fieldsUsed internally to generate a list of property to extract from the ad.get_seriesFetch a list page from the target website.has_issueTrue if we met issues with this ad before.is_scrapedTrue if we already scraped this ad before.okPrint out an success message.prepareJust before saving the values.runRun the scrapper.transform_pageTransform HTML content of the series page before parsing it.Start a migrationUseYoyo:yoyo new ./migrations -m "Your migration's description"And apply it:yoyo apply --database mysql://user:password@host/db ./migrations