Package: auction-scraper
Auction ScraperScrape auction data auction sites into a sqlite databaseCurrently supports: catawiki, ebay, liveauctioneersCan be used as a CLI tool, or interfaced with directlyInstallationYou caninstall with pip:pip install auction-scraperNew backend supportWant to scrape an auction house not listed above?  Fear not - through our partnership withDreaming Spires, you can request that we build additional backend scrapers to extend the functionality.  Emailcontact@dreamingspires.devfor more info.We also accept PRs, so feel free to write your own backend and submit it, if you require.  Instructions for this can be found under theBuilding new backendssection.Usageauction-scraperwill scrape data from auctions, profiles, and searches on the specified auction site.  Resulting textual data is written to asqlite3database, with images and backup web pages optionally being written to adata directory.The tool is invoked as:Usage: auction-scraper [OPTIONS] DB_PATH BACKEND:[ebay|liveauctioneers]
                       COMMAND [ARGS]...

Options:
  DB_PATH                         The path of the sqlite database file to be
                                  written to  [required]

  BACKEND:[ebay|liveauctioneers]  The auction scraping backend  [required]
  --data-location TEXT            The path additional image and html data is
                                  saved to

  --save-images / --no-save-images
                                  Save images to data-location.  Requires
                                  --data-location  [default: False]

  --save-pages / --no-save-pages  Save pages to data-location. Requires
                                  --data-location  [default: False]

  --verbose / --no-verbose        [default: False]
  --base-uri TEXT                 Override the base url used to resolve the
                                  auction site

  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.

Commands:
  auction  Scrapes an auction site auction page.
  profile  Scrapes an auction site profile page.
  search   Performs a search, returning the top n_results results for each...Auction modeIn auction mode, an auction must be specified as either a uniqueauction IDor as a URL.  The textual data is scraped into the[BACKEND]_auctionstable ofDB_PATH, the page is scraped into[data-location]/[BACKEND]/auctions, and the images into[data-location]/[BACKEND]/images.  The--base-urloption determines the base URL from which to resolveauction IDs,profile IDs, and searchquery stringsif specified, otherwise defaulting to the default for the specified backend.Example usage:# Scraping an auction by URLauction-scraperdb.dbliveauctioneersauctionhttps://www.liveauctioneers.com/item/88566418_cameroon-power-or-reliquary-figure# Equivalently scraping from an auction IDauction-scraperdb.dbliveauctioneersauction88566418# Scraping an auction, including all images and the page itself, into data-locationauction-scraper--data-location=./data--save-images--save-pagesdb.dbliveauctioneersauction88566418Profile modeIn profile mode, a profile must be specified as either a uniqueuser IDor as a URL.  The textual data is scraped into the[BACKEND]_profilestable ofDB_PATH, and the page is scraped into[data-location]/[BACKEND]/profiles.  The--base-urloption determines the base URL from which to resolveauction IDs,profile IDs, and searchquery stringsif specified, otherwise defaulting to the default for the specified backend.Example usage:# Scraping a profile by URLauction-scraperdb.dbliveauctioneersprofilehttps://www.liveauctioneers.com/auctioneer/197/hindman/# Equivalently scraping from a profile IDauction-scraperdb.dbliveauctioneersauction197# Scraping a profile, including the page itself, into data-locationauction-scraper--data-location=./data--save-pagesdb.dbliveauctioneersprofile197Search modeIn search mode, at least oneQUERY_STRINGmust be provided alongsideN_RESULTS.  It will scrape the auctions pertaining to the topN_RESULTSresults from theQUERY_STRING.  The--base-urloption determines the base URL from which to resolve the search if specified, otherwise defaulting to the default for the specified backend.Example usage:# Search one result by a single search termauction-scraperdb.dbsearch1"mambila art"# Search ten results by two search terms, scraping images and pages into data-locationauction-scraper--data-location=./data--save-images--save-pagesdb.dbsearch10"mambila""mambilla"Running continuously using systemdauction-scraper@.serviceandauction-scraper@.timer, once loaded by systemd, can be used to schedule the running ofauction-scraperwith user-given arguments according to a schedule.Running as a systemd root serviceCopyauction-scraper@.serviceandauction-scraper@.timerto/etc/systemd/system/.Modifyauction-scraper@.timerto specify the schedule you require.Reload the system daemons.  As root:systemctldaemon-reloadRun (start now) and enable (restart on boot) the systemd-timer, specifying the given arguments, within quotes, after the '@'.  For example, as root:systemctlenable--nowauction-scraper@"db.db liveauctioneers search 10 mambila".timerFind information about your running timers with:systemctllist-timersStop your currently running timer with:systemctlstopauction-scraper@"db.db liveauctioneers search 10 mambila".timerDisable your currently running timer with:systemctldisableauction-scraper@"db.db liveauctioneers search 10 mambila".timerA new timer is created for each unique argument string, so the arguments must be specified when stopping or disabling the timer.Some modification may be required to run as a user service, including placing the service and timer files in~/.local/share/systemd/user/.Building from sourceEnsure poetry isinstalled.  Then from this directory install dependencies into the poetry virtual environment and build:poetryinstall
poetrybuildSource and wheel files are built intoauction_scraper/dist.Install it across your user withpip, outside the venv:cd./dist
python3-mpipinstall--user./auction_scraper-0.0.1-py3-none-any.whlorcd./dist
pipinstall./auction_scraper-0.0.1-py3-none-any.whlRunauction-scraperto invoke the utility.Interfacing with the APIEach backend ofauction-scrapercan also be invoked as a Python library to automate its operation.  The backends implement the abstract classauction_scraper.abstract_scraper.AbstractAuctionScraper, alongside the abstract SQLAlchemy modelsauction_scraper.abstract_models.BaseAuctionandauction_scraper.abstract_models.BaseProfile.
The resulting scraper exposes methods to scrape auction, profile, and search pages into these SQLAlchemy model objects, according to the following interface:def scrape_auction(self, auction, save_page=False, save_images=False):
    """
    Scrapes an auction page, specified by either a unique auction ID
    or a URI.  Returns an auction model containing the scraped data.
    If specified by auction ID, constructs the URI using self.base_uri.
    If self.page_save_path is set, writes out the downloaded pages to disk at
    the given path according to the naming convention specified by
    self.auction_save_name.
    Returns a BaseAuction
    """def scrape_profile(self, profile, save_page=False):
    """
    Scrapes a profile page, specified by either a unique profile ID
    or a URI.  Returns an profile model containing the scraped data.
    If specified by profile ID, constructs the URI using self.base_uri.
    If self.page_save_path is set, writes out the downloaded pages to disk at
    the given path according to the naming convention specified by
    self.profile_save_name.
    Returns a BaseProfile
    """def scrape_search(self, query_string, n_results=None, save_page=False,
        save_images=False):
    """
    Scrapes a search page, specified by either a query_string and n_results,
    or by a unique URI.
    If specified by query_string, de-paginates the results and returns up
    to n_results results.  If n_results is None, returns all results.
    If specified by a search_uri, returns just the results on the page.
    Returns a dict {auction_id: SearchResult}
    """def scrape_auction_to_db(self, auction, save_page=False, save_images=False):
    """
    Scrape an auction page, writing the resulting page to the database.
    Returns a BaseAuction
    """def scrape_profile_to_db(self, profile, save_page=False):
    """
    Scrape a profile page, writing the resulting profile to the database.
    Returns a BaseProfile
    """def scrape_search_to_db(self, query_strings, n_results=None, \
        save_page=False, save_images=False):
    """
    Scrape a set of query_strings, writing the resulting auctions and profiles
    to the database.
    Returns a tuple ([BaseAuction], [BaseProfile])
    """Building new backendsAll backends live ataction_scraper/scrapersin their own specific directory.  It should implement the abstract classauction_scraper.abstract_scraper.AbstractAuctionScraperin a filescraper.py, and the abstract SQLAlchemy modelsauction_scraper.abstract_models.BaseAuctionandauction_scraper.abstract_models.BaseProfileinmodels.py.TheAuctionScraperclass must extendAbstractAuctionScraperand implement the following methods:# Given a uri, scrape the auction page into an auction object (of type BaseAuction)def_scrape_auction_page(self,uri)# Given a uri, scrape the profile page into an profile object (of type BaseAuction)def_scrape_profile_page(self,uri)# Given a uri, scrape the search page into a list of results (of type {auction_id: SearchResult})def_scrape_search_page(self,uri)It must also supply defaults to the following variables:auction_tableprofile_tablebase_uriauction_suffixprofile_suffixsearch_suffixbackend_nameAuthorsEdd Salkieldedd@salkield.uk- Main codebaseMark Todd                       - Liveauctioneers scraperJonathan Tanner                 - Catawiki scraper