Package: hemnes
HEMNESReport Bug·Request FeatureTable of ContentsAbout the ProjectBuilt WithWhy Selenium or Why not Python RequestsGetting StartedRuntimeInstallationBasic UsageProduct Object DetailsAdvanced UsageSpecifying the path to ChromeDriverRequired Keywords & Strict SearchLoggingRetrieving a Specific Number of ResultsSpeeding UpUsing the Tag FieldWriting Results to JSON/CSVTestsWhat's NextRelease HistoryLicenseAbout the ProjectHemnes is a pip package for scraping product data from ikea, built becauserewriting scrapers is a waste of time. Hemnes gets you to the point of being able to query Ikea's product catalog in less than 30 seconds (I didn't time this).The following product data is collected by Hemnes:name (str)- name of the productid (str)- unique product idprice (float)- product priceurl (str)- url to product pagerating (float)- average customer ratingimg_urls (list[str])- urls to product imagescolors (list[str])- product colors (seeCOLORSinhemnes/helpers/find_elementfor a full list of colors being searched for)Hemnes comes equipped with a bit of helpful functionality for doing things like:finding products with specific keywords in their descriptionsloggingwriting results to JSONRead more about extended functionality hereBuilt WithBeautifulSoupChromeDriverPyTestSelenium WebDriverWhy Selenium or Why not Python RequestsIf you do a quick browse through stack overflow posts about scraping ikea - or if you are considering writing a scraper yourself - you will probably come across people using selenium Webdriver for pages that don't need it. Selenium is heavier than python requests, however,webdriver can load angular generated content whereas requests cannot.Ikea's current website uses angular for its search results. This is a change from its older search, which was accessible via python requests. The new search provides more accurate results, and less garbage. Speaking from experience, the old search is really terrible and provides a number of trash results for any given query that I had trouble connecting back to my original search (one example of this is the old search returning 58 pages of results for 'table', which included things like placemats and dog toys; the new search returns 15, and all of the results are actually tables)Getting StartedHemnes is pretty straightforward to install and use. It functions exactly as you would expect a web-scraping package to function - enter a query and get results back.RuntimePython 3+InstallationHemnes is installed using standard pip installation. To install from PyPI runpip3installhemnesAlternatively, you can clone the repo and then run pip install inside the directory# clone the repositorygitclonehttps://github.com/sayeefrmoyen/hemnes.gitcdhemnes
pip3install.# install the current directory as a packageChromeDriverHemnes usesChromeDriverto load webpages.  If ChromeDriver is already installed and on your syspath you can skip the rest of this section.To use ChromeDriver you will need toinstall Google Chrome. If you already have google chrome installedI believe that you should be able to run Hemnes without a problem. For full disclosure, I am not familiar with the google chrome or chrome driver code, and do not know how they interact, but I have had no issues running Hemnes without ever explicitly installing the ChromeDriver binary.With that said, Selenium's documentation for WebDriver suggests that you install both a version of Google Chrome and ChromeDriver. If you find that after installing Google Chrome you are still receiving errors regarding ChromeDriver, then you should proceed toinstall ChromeDriver.At this point, you should be ready to start using Hemnes.Basic UsageAt its simplest, Hemnes only requires a query. Use theprocess_queryfunction to retrieve product data for a given query.process_queryreturns alist[Product]containing the products matching the given query.importhemnes# query ikea's product catalog for products tagged as chairs# chair_results is now a list[Product] containing all of the products# in ikea's catalog of chairsresults=hemnes.process_query('chair')Product Object DetailsHemnes returns query results in the form of alist[Product].Productis a helper class containing the following fields:name (str)- name of the productid (str)- unique product idprice (float)- product priceurl (str)- url to product pagerating (float)- average customer ratingimg_urls (list[str])- urls to product imagescolors (list[str])- product colorstag (str)- flexible usage fieldMost of these are rather self-explanatory; I'm only going to talk about thetagfield in depth. Thetagattribute is specified for all products returned by a single call toprocess_query, and is included for flexible use. One example of why such a field would be useful is if you were storing this data in a database and needed a primary key to search on - you could usetagto indicate the type of product (chair, table, etc.). By defaulttagis set toNone. For more details on using tag, seesetting up additional optionsOne more thing to note is that all of these fields, excludingurl, have the potential to not be found, and subsequently being set toNone. However, based on testing and examining the structure of Ikea's product webpages, it is unlikely that any of these attributes are unable to be found. The one exception to this rule is theratingfield, which will be set toNoneany time the product has no reviews.Advanced UsageHemnes expects to be passed anOptionsobject to specify a number of additional settings.Optionsis a helper class for organizing passing a large number of parameters toprocess_query. If you don't provide anOptionsobject toprocess_query, a number of default settings are selected. The rest of this section will discuss how to modify those settings.Specifying the path to the ChromeDriver binaryIf you have installed the ChromeDriver binary and Google Chrome browser, but are still encountering errors when running Hemnes, you may need to explicitly pass the path to the binary to selenium Webdriver. To do so, use theOptionsclassimporthemnes...# explicitly passing the chromedriver binary to webdriveroptions=hemnes.Options()options.cdriver_path='path/to/chromedriver/binary'results=hemnes.process_query('your query',options)Required Keywords & Strict SearchingSometimes it's necessary to refine your query beyond usual high-level query terms. Hemnes allows you to specify a number ofkeywordsto search for on product pages, and only return products which contain all or some of those words.Optionsaccepts setting thekeywordsfield to aset[str]containing the desired keywords.You can also specify whether or not all of the keywords should be required by settingOptionsstrictfield to abool. By default,strictis set toFalse, meaning that ifkeywordsare passed, any product with at least one keyword will be returned. To require that allkeywordsare found for returned products, setstricttoTrueimporthemnes...# setting required keywordsoptions=hemnes.Options()options.keywords={'large','comfortable'}# enable strict-searching, requiring all products to contain all of the# keywords in order to be returned. If disabled or untouched, any# product with at least one of the keywords will be returnedoptions.strict=Trueresults=hemnes.process_query('chair',options)For those who are curious about wherekeywordsare being looked for, Hemnes searches through 3 different product description sections on each product's page forkeywords.Enable LoggingSome jobs for queries that return a large number of results may take a while to complete. Even for shorter jobs, it can be helpful to see where Hemnes is in processing a query. In order to enable logging process results, set thelogfield inOptionstoTrue. By defaultlogis set toFalseto avoid overwhelming unsuspecting users.importhemnes...# enabling logsoptions=hemnes.Options()options.log=True# enable logging - this will log to both stdout and to a# logfile found at 'hemnes-logs/hemnes-MONTH-DAY-HOUR-MINUTE-SECOND.logresults=hemnes.process_query('chair',options)By default logging will log to both stdout and to a log file that will look something likehemnes-logs/hemnes-04-23-02:41:16.log- the file is namedhemnes-MONTH-DAY-HOUR-MINUTE-SECOND.log. If there is nohemnes-logsdirectory it will be created prior to trying to write files to it.Hemnes will log things like:when a valid product is foundwhen an invalid product is found (e.g. fails keyword requirements)total number of valid products to be returnedany potential errorsRetrieving a Specific Number of ResultsIf you only need a specific number of results for a given query, set thenum_resultsfield ofOptions.importhemnes...# setting a target number of resultsoptions=hemnes.Options()options.num_results=10# hemnes will only return up to 10 productsresults=hemnes.process_query('chair',options)Hemnes will return the number of products requested, or fewer if the query did not return enough results.Speeding UpLoading angular pages can be slow, mostly because it takes time to retrieve full DOM from such pages. Altering the imposed sleep time for DOM to be fully loaded can drastically increase the speed of Hemnes.By default, Hemnes requires a 3 second sleep time after each page request in order to insure that the DOM is fully loaded. For users with fast download speeds, this may be longer than is necessary. In order to reduce the sleep time after network requests, set thesleep_timeattribute ofOptionsto something more appropriate for your internet connection.importhemnes...# setting required keywordsoptions=hemnes.Options()options.sleep_time=1# hemnes will now wait only 1 second for DOM to be loaded for# newly retrieved pages. Users should set sleep_time to an# appropriate amount of time based on their internet connection.# The default setting of 3-seconds should be fine for almost all usersresults=hemnes.process_query('chair',options)Using Product's Tag FieldThetagfield can be set for all returnedProductfor a given call toprocess_queryby setting it inOptions.tagshould be of typestr.importhemnes...# setting the tag attributeoptions=hemnes.Options()options.tag='chair'# all of the products returned in results will have their tag field# set to the string "chair"results=hemnes.process_query('chair',options)Writing Results to JSON or CSVStoring scraped results to JSON or CSV can be done by setting the appropriate paths to the files usingOptions.
Hemnes will delete an existing file if the path already exists.Note: anyNonefields will be written asnullin JSON, or left empty in CSVimporthemnes...options=hemnes.Options()# setting the path to a JSON dump fileoptions.json_dump='path_to_json_file.json'# setting the path to a CSV dump fileoptions.csv_dump='path_to_csv_file.csv'# results will be written to the specified json or csv filesresults=hemnes.process_query('chair',options)TestsTests are broken intotest_isolatedandtest_integrated.test_isolatedare, as they sound, true unit-tests for individual functionality.test_integratedare tests for higher-level functionality that depends on some of the lower level functionality provided by functions tested intest_isolated.Note:test_integratedtake a bit of time to execute as they go through the entire process of retrieving product data from start to finishTo run the tests clone the repository and run usingpytest# install pytest if you do not have pytest installedpip3installpytest# clone the repositorygitclonehttps://github.com/sayeefrmoyen/hemnes.gitcdhemnes# --verbose flag is optional - provides helpful console outputpytest--verbose# run all of the testspytesttest/test_isolated.py--verbose# only run isolation testspytesttest/test_integrated.py--verbose# only run intregration testsWhat's NextI'm ok with where this is now - if anyone requests more features I will look into them, but otherwise I will be maintaining hemnes as-is.Release History1.2 - add saving to csv1.1 - add saving to json1.0 - first stable, tested release0.*.* - pre-test releasesLicenseDistributed under the MIT License. See LICENSE for more information.