Package: transistor
Web data collection and storage for intelligent use cases.transistorAboutThe web is full of data. Transistor is a web scraping framework for collecting, storing, and using targeted data from structured web pages.Transistor’s current strengths are in being able to:provide an interface to useSplashheadless browser / javascript rendering service.includesoptionalsupport for using the scrapinghub.comCrawlera‘smart’ proxy service.ingest keyword search terms from a spreadsheet or use RabbitMQ or Redis as a message broker, transforming keywords into task queues.scale oneSpiderinto an arbitrary number of workers combined into aWorkGroup.coordinate an arbitrary number ofWorkGroupssearching an arbitrary number of websites, into one scrape job.send out all theWorkGroupsconcurrently, using gevent based asynchronous I/O.return data from each website for each search term ‘task’ in our list, for easy website-to-website comparison.export data to CSV, XML, JSON, pickle, file object, and/or your own custom exporter.save targeted scrape data to the database of your choice.Suitable use cases include:comparing attributes like stock status and price, for a list ofbook titlesorpart numbers, across multiple websites.concurrently process a large list of search terms on a search engine and then scrape results, or follow links first and then scrape results.Development of Transistor is sponsored byBOM Quote Manufacturing. Here is a Medium story from the author about creating Transistor:That time I coded 90-hours in one week.Primary goals:Enable scraping targeted data from a wide range of websites including sites rendered with Javascript.Navigate websites which present logins, custom forms, and other blockers to data collection, like captchas.Provide asynchronous I/O for task execution, usinggevent.Easily integrate within a web app likeFlask,Django, or other python basedweb frameworks.Provide spreadsheet based data ingest and export options, like import a list of search terms from excel, ods, csv, and export data to each as well.Utilize quick and easy integrated task work queues which can be automatically filled with data search terms by a simple spreadsheet import.Able to integrate with more robust task queues likeCelerywhile usingrabbitmqorredisas a message broker as desired.Provide hooks for users to persist data via any method they choose, while also supporting our own opinionated choice which is aPostgreSQLdatabase along withnewt.db.Contain useful abstractions, classes, and interfaces for scraping and crawling with machine learning assistance (wip, timeline tbd).Further support data science use cases of the persisted data, where convenient and useful for us to provide in this library (wip, timeline tbd).Provide a command line interface (low priority wip, timeline tbd).QuickstartFirst, installTransistorfrom pypi:pip install transistorIf you have previously installedTransistor, please ensure you are using the latest version:pip-install --upgrade transistorNext, setup Splash, following the Quickstart instructions. Finally, follow the minimal abbreviated Quickstart examplebooks_to_scrapeas detailed below.This example is explained in more detail in the source code found in theexamples/books_to_scrapefolder, including fully implementing object persistence withnewt.db.Quickstart: Setup SplashSuccessfully scraping is now a complex affair. Most websites with useuful data will rate limit, inspect headers, present captchas, and use javascript that must be rendered to get the data you want.This rules out using simple python requests scripts for most serious use. So, setup becomes much more complicated.To deal with this, we are going to useSplash,
“A Lightweight, scriptable browser as a service with an HTTP API”.Transistor also supports theoptionaluse of asmart proxy servicefromscrapinghubcalledCrawlera.
The crawlera smart proxy service helps us:avoid getting our own server IP bannedenable regional browsing which is important to us, because data can differ per region on the websites we want to scrape, and we are interested in those differencesThe minimum monthly cost for the smallest size crawleraC10plan is $25 USD/month. This level is useful but can easily be overly restrictive.  The next level up is $100/month.The easiest way to get setup with Splash is to useAquariumand that is what we are going to do. Using Aquarium requires Docker and Docker Compose.Windows SetupOn Windows, the easiest way to get started with Docker is to useChocolatelyto install docker-desktop (the successor to docker-for-windows, which has now been depreciated). Using Chocolately requiresinstalling Chocolately.Then, to install docker-desktop with Chocolately:C:\> choco install docker-desktopYou will likely need to restart your Windows box after installing docker-desktop, even if it doesn’t tell you to do so.All PlatformsInstall Docker for your platform. For Aquarium, follow theinstallation instructions.After setting up Splash with Aquarium, ensure you set the following environment variables:SPLASH_USERNAME='<username you set during Aquarium setup>'SPLASH_PASSWORD='<password you set during Aquarium setup>'Finally, to run Splash service,cd to the Aquarium repo on your hard drive, and then rundocker-composeupin your command prompt.Troubleshooting Aquarium and Splash service:Ensure you are in theaquariumfolder when you run thedocker-composeupcommand.You may have some initial problem if you did not share your hard drive with Docker.Share your hard drive with docker (google is your friend to figure out how to do this).Try to run thedocker-composeupcommand again.Note, upon computer/server restart, you need to ensure the Splash service is started, either daemonized or withdocker-composeup.At this point, you should have a splash service running in your command prompt.CrawleraUsing crawlera is optional and not required for thisbooks_to_scrapequickstart.But, if you want to use Crawlera with Transistor, first, register for the service and buy a subscription atscrapinghub.com.After registering for Crawlera, create accounts in scrapinghub.com for each region you would like to present a proxied ip address from. For our case, we are setup to handle three regions, ALL for global, China, and USA.Next, you should set environment variables on your computer/server with the api key for each region you need, like below:CRAWLERA_ALL='<your crawlera account api key for ALL regions>'CRAWLERA_CN='<your crawlera account api key for China region>'CRAWLERA_USA='<your crawlera account api key for USA region>'CRAWLERA_REGIONS='CRAWLERA_ALL,CRAWLERA_USA,CRAWLERA_CN'There are some utility functions which are helpful for working with crawlera found intransistor/utility/crawlera.pywhich require theCRAWLERA_REGIONSenvironment variable to be set.CRAWLERA_REGIONSshould just be a comma separated string of whatever region environment variables you have set.Finally, to use Crawlera, you will need to pass a keyword arg likecrawlera_user=<yourapi key>into your custom Scraper spider which has been subclassed from theSplashScraperclass.
Alternately, you can directly setcrawlera_userin your custom subclassed Scraper’s__init__()method likeself.crawlera_user =os.environ.get('CRAWLERA_USA',None).Last, you must pass in a Lua script in thescriptargument which supports the Crawlera service. We have included two Lua scripts intransistor\scrapers\scriptsfolder which will be helpful to work out-of-the-box.
Of course, to get the full power of Splash + Crawlera you will need to read their documentations and also come up to speed on how to customize the Lua script to fully use Splash, to do things like fill out forms and navigate pages.Quickstart:books_to_scrapeexampleSeeexamples/books_to_scrapefor a fully working example with more detailed notes in the source code.  We’ll go through an abbreviated setup here, without many of the longer notes and database/persistence parts that you can find in theexamplesfolder source code.In this abbreviated example, we will create aSpiderto crawl the books.toscrape.com website to search for 20 different book titles, which the titles are ingested from an excel spreadsheet. After we find the book titles, we will export the targeted data to a different csv file.Thebooks_to_scrapeexample assumes we have a column of 20 book titles in an excel file, with a column heading in the spreadsheet nameditem.  We plan to scrape the domainbooks.toscrape.comto find the book titles. For the book titles we find, we will scrape the sale price and stock status.First, let’s setup a custom scraper Spider by subclassingSplashScraper. This will enable it to use the Splash headless browser.Next, create a few custom methods to parse the html found by theSplashScraperand saved in theself.pageattribute, with beautifulsoup4.fromtransistor.scrapersimportSplashScraperclassBooksToScrapeScraper(SplashScraper):"""
    Given a book title, scrape books.toscrape.com/index.html
    for the book cost and stock status.
    """def__init__(self,book_title:str,script=None,**kwargs):"""
        Create the instance with a few custom attributes and
        set the baseurl
        """super().__init__(script=script,**kwargs)self.baseurl='http://books.toscrape.com/'self.book_title=book_titleself.price=Noneself.stock=Nonedefstart_http_session(self,url=None,timeout=(3.05,10.05)):"""
        Starts the scrape session. Normally, you can just call
        super().start_http_session(). In this case, we also want to start out
        with a call to self._find_title() to kickoff the crawl.
        """super().start_http_session(url=url,timeout=timeout)returnself._find_title()# now, define your custom books.toscrape.com scraper logic belowdef_find_title(self):"""
        Search for the book title in the current page. If it isn't found, crawl
        to the next page.
        """ifself.page:title=self.page.find("a",title=self.book_title)iftitle:returnself._find_price_and_stock(title)else:returnself._crawl()returnNonedef_next_page(self):"""
        Find the url to the next page from the pagination link.
        """ifself.page:next_page=self.page.find('li',class_='next').find('a')ifnext_page:ifnext_page['href'].startswith('catalogue'):returnself.baseurl+next_page['href']else:returnself.baseurl+'/catalogue/'+next_page['href']returnNonedef_crawl(self):"""
        Navigate to the next url page using the SplashScraper.open() method and
        then call find_title again, to see if we found our tasked title.
        """ifself._next_page():self.open(url=self._next_page())returnself._find_title()returnprint(f'Crawled all pages. Title not found.')def_find_price_and_stock(self,title):"""
        The tasked title has been found and so now find the price and stock and
        assign them to class attributes self.price and self.stock for now.
        """price_div=title.find_parent("h3").find_next_sibling('div',class_='product_price')self.price=price_div.find('p',class_='price_color').textself.stock=price_div.find('p',class_='instock availability').text.translate({ord(c):Noneforcin'\n\t\r'}).strip()print('Found the Title, Price, and Stock.')Next, we need to setup two more subclasses from baseclassesSplashScraperItemandItemLoader. This will allow us to export the data from theSplashScraperspider to the csv spreadsheet.Specifically, we are interested to export thebook_title,stockandpriceattributes. See more detail inexamples/books_to_scrape/persistence/serialization.pyfile.fromtransistor.persistence.itemimportFieldfromtransistor.persistenceimportSplashScraperItemsfromtransistor.persistence.loaderimportItemLoaderclassBookItems(SplashScraperItems):# -- names of your customized scraper class attributes go here -- #book_title=Field()# the book_title which we searchedprice=Field()# the self.price attributestock=Field()# the self.stock attributedefserialize_price(value):"""
    A simple serializer used in BookItemsLoader to ensure USD is
    prefixed on the `price` Field, for the data returned in the scrape.
    :param value: the scraped value for the `price` Field
    """ifvalue:returnf"UK{str(value)}"classBookItemsLoader(ItemLoader):defwrite(self):"""
        Write your scraper's exported custom data attributes to the
        BookItems class. Call super() to also capture attributes
        built-in from the Base ItemLoader class.

        Last, ensure you assign the attributes from `self.items` to
        `self.spider.<attribute>` and finally you must return
        self.items in this method.
        """# now, define your custom itemsself.items['book_title']=self.spider.book_titleself.items['stock']=self.spider.stock# set the value with self.serialize_field(field, name, value) as needed,# for example, `serialize_price` below turns '£50.10' into 'UK £50.10'# the '£50.10' is the original scraped value from the website stored in# self.scraper.price, but we think it is more clear as 'UK £50.10'self.items['price']=self.serialize_field(field=Field(serializer=serialize_price),name='price',value=self.spider.price)# call super() to write the built-in SplashScraper Items from ItemLoadersuper().write()returnself.itemsFinally, to run the scrape, we will need to create a main.py file.  This is all we need for the minimal example to scrape and export targeted data to csv.So, at this point, we’ve:Setup a custom scraperBooksToScrapeScraperby subclassingSplashScraper.SetupBookItemsby subclassingSplashScraperItems.SetupBookItemsLoaderby subclassingItemLoader.Wrote a simpleserializerwith theserialize_pricefunction, which appends ‘UK’ to the returnedpriceattribute data.Next, we are ready to setup amain.pyfile as the final entry point to run our first scrape and export the data to a csv file.The first thing we need to do is perform some imports.#  -*- coding: utf-8 -*-# in main.py, monkey patching for gevent must be done firstfromgeventimportmonkeymonkey.patch_all()# you probably need to add your project directory to the pythonpath like belowimportsyssys.path.insert(0,"C:/Users/<username>/repos/books_to_scrape")# finally, import from transistor and your own custom codefromtransistorimportStatefulBook,WorkGroup,BaseWorkGroupManagerfromtransistor.persistence.exportersimportCsvItemExporterfrom<path-to-your-custom-scraper>importBooksToScrapeScraperfrom<path-to-your-custom-Items/ItemsLoader>importBookItems,BookItemsLoaderSecond, setup aStatefulBookwhich will read thebook_titles.xlsxfile and transform the book titles from the spreadsheet “titles” column into task queues for ourWorkGroups.# we need to get the filepath to your book_titles.xlsx excel file, you can copy it# from transistor/examples/books_to_scrape/schedulers/stateful_book/book_titles.xlsx# need a variable like below:# filepath = 'your/path/to/book_titles.xlsx'# including some file path code here as a hint because it's not so straightforwardfrompathlibimportPathfromos.pathimportdirnameasdfromos.pathimportabspathroot_dir=d(d(abspath(__file__)))defget_file_path(filename):"""
    Find the book_titles excel file path.
    """root=Path(root_dir)filepath=root/'files'/filenamereturnr'{}'.format(filepath)# now we can use get_file_path to set the variable named `filepath`filepath=get_file_path('book_titles.xlsx')trackers=['books.toscrape.com']tasks=StatefulBook(filepath,trackers,keywords="titles")Third, setup a list of exporters which than then be passed to whicheverWorkGroupobjects you want to use them with.  In this case, we are just going to use the built-inCsvItemExporterbut we could also use additional exporters to do multiple exports at the same time, if desired.exporters=[CsvItemExporter(fields_to_export=['book_title','stock','price'],file=open('c:/book_data.csv','a+b'))]Fourth, setup theWorkGroupin a list we’ll callgroups. We use a list here because you can setup as manyWorkGroupobjects with unique target websites and as many individual workers, as you need:groups=[WorkGroup(name='books.toscrape.com',url='http://books.toscrape.com/',spider=BooksToScrapeScraper,items=BookItems,loader=BookItemsLoader,exporters=exporters,workers=20,# this creates 20 Spiders and assigns each a book as a taskkwargs={'timeout':(3.0,20.0)})]Fifth, setup theWorkGroupManagerand prepare the file to call themanager.main()method to start the scrape job:# If you want to execute all the scrapers at the same time, ensure the pool is# marginally larger than the sum of the total number of workers assigned in the# list of WorkGroup objects. However, sometimes you may want to constrain your pool# to a specific number less than your scrapers. That's also OK. This is useful# like Crawlera's C10 instance, only allows 10 concurrent workers. Set pool=10.manager=BaseWorkGroupManager(job_id='books_scrape',tasks=tasks,workgroups=groups,pool=25)if__name__=="__main__":manager.main()# call manager.main() to start the job.Finally, runpython main.pyand thenprofit. After a brief Spider runtime to crawl the books.toscrape.com website and write the data, you should have a newly exported csv file in the filepath you setup, ‘c:/book_data.csv’ in our example above.To summarize what we did inmain.py:We setup aBaseWorkGroupManager, wrapped our spiderBooksToScrapeScraperinside a list ofWorkGroupobjects calledgroups. Then we passed thegroupslist to theBaseWorkGroupManager.Passing a list ofWorkGroupobjects allows theWorkGroupManagerto run multiple jobs targeting different websites, concurrently.In this simple example, we are only scrapingbooks.toscrape.com, but if we wanted to also scrapebooks.toscrape.com.cn, then we’d setup twoBaseGroupobjects and wrap them each in their ownWorkGroup, one for each domain.NOTE-1: A more robust use case will also subclass theBaseWorkerclass. Because, it provides several methods as hooks for data persistence and post-scrape manipulation.
Also, one may also consider to subclass theWorkGroupManagerclass and override it’smonitormethod. This is another hook point to have access to theBaseWorkerobject before it shuts down for good.Refer to the full example in theexamples/books_to_scrape/workgroup.pyfile for an example of customizingBaseWorkerandWorkGroupManagermethods. In the example, we show how to to save data to postgresql with newt.db but you can use whichever db you choose.NOTE-2: If you do try to follow the more detailed example  inexamples/books_to_scrape, including data persistence with postgresql and newt.db, you may need to set the environment variable:TRANSISTOR_DEBUG=1Whether or not you actually need to set thisTRANSISTOR_DEBUGenvironment variable will depend on how you setup your settings.py and newt_db.py files.
If you copy the files verbatim as shown in theexamples/books_to_scrapefolder, then you will need to set it.Directly Using A SplashScraperPerhaps you just want to do a quick one-off scrape?It is possible to just use your custom scraper subclassed fromSplashScraperdirectly, without going through all the work to setup aStatefulBook,BaseWorker,BaseGroup,WorkGroup, andWorkGroupManager.Just fire it up in a python repl like below and ensure thestart_http_sessionmethod is run, which can generally be done by settingautorun=True.>>>frommy_custom_scrapers.component.mousekeyimportMouseKeyScraper>>>ms=MouseKeyScraper(part_number='C1210C106K4RACTU',autorun=True)After the scrape completes, various methods and attributes fromSplashScraperandSplashBrowserare available, plus your custom attributes and methods from your own subclassed scraper, are available:>>>print(ms.stock())'4,000'>>>print(ms.pricing())'{"1" : "USD $0.379", "10" : "USD $0.349"}'Architecture SummaryTransistor provides useful layers and objects in the following categories:Layers & Servicesjavascript rendering service / headless browser layer:Transistor usesSplashimplemented withAquariumcookicutter docker template.Splash provides a programmable headless browser to render javascript and Aquarium provides robust concurrency with multiple Splash instances that are load balanced withHAProxy.Transistor provides integration with Splash through ourSplashBrowserclass found intransistor/browsers/splash_browser.py.smart proxy service:Transistor supports use ofCrawlera, which is a paidsmart proxy serviceproviding robust protection against getting our own ip banned while scraping sites that actively present challenges to web data collection.Crawlera use is optional. It has a minimum monthly cost of $25 USD for starter package and next level up is currently $100 USD/month.in using Crawlera, the concurrency provided by gevent for asynchronous I/O along with Splash running with Aquarium, is absolutely required, because a single request with Splash + Crawlera is quite slow, taking up to15 minutesor more to successfully return a result.Spidersbrowserssee:transistor/browserswrappython-requestsandbeautifulsoup4libraries to serve our various scraping browser needs.browser API is generally created by subclassing and overriding the well knownmechanicalsouplibrary to work with Splash and/or Splash + Crawlera.if Javascript support is not needed for a simple scrape, it is nice to just use mechanicalsoup’sStatefulBrowserclass directly as a Scraper, like as shown inexamples/cny_exchange_rate.py.aBrowserobject is generally instantiated inside of aScraperobject, where it handles items like fetching the page, parsing headers, creating aself.pageobject to parse with beautifulsoup4, handling failures with automatic retries, and setting class attributes accessible to ourScraperobject.scrapersseetransistor/scrapersinstantiates a browser to grab thepageobject, implements various html filter methods onpageto return the target data, can use Splash headless browser/javascript rendering service to navigate links, fill out forms, and submit data.for a Splash or Splash + Crawlera based scraperSpider, theSplashScraperbase class provides a minimal required Lua script and all required connection logic. However, more complex use cases will require providing your own custom modified Lua script.the scraper design is built around gevent based asynchronous I/O, and this design allows to send out an arbitrarily large number of scraper workers, with each scraper worker assigned a specific scrape task to complete.the current core design, in allowing to send out an arbitrarily large number of scraper workers, is not necessarily an optimal design to ‘crawl’ pages in search of targeted data. Where it shines is when you need to use a webpage search function on an arbitrarily large list of search tasks, await the search results for each task, and finally return a scraped result for each task.crawlers(wip, on the to-do list)seetransistor/crawlers(not yet implemented)this crawlingSpiderwill be supported through a base class calledSplashCrawler.while it is straightforward to use the current Transistor scraperSplashScraperdesign to do basic crawling (seeexamples/books_to_scrape/scraper.pyfor an example) the current way to do this with Transistor is not optimal for crawling. So we’ll implement modified designs for crawling spiders.specifics TBD, may be fully custom or else may reuse some good architecture parts ofscrapy, although if we do that, it will be done so we don’t need a scrapy dependency and further it will be using gevent for asynchronous I/O.Program I/Oschedulers:BOOKSseetransistor/schedulers/booksaStatefulBookobject provides an interface to work with spreadsheet based data.for example, a book facilitates importing a column of keyword search term data, like ‘book titles’ or ‘electronic component part numbers’, from a designated column in an .xlsx file.after importing the keyword search terms, the book will transform each search term into a task contained in aTaskTrackerobjecteachTaskTrackerwill contain a queue of tasks to be assigned by theWorkGroupManager, and will ultimately allow an arbitrarily large number ofWorkGroupsofBaseWorkersto execute the tasks, concurrently.RabbitMQ & Redisseetransistor/schedulers/brokersprovides theExchangeQueueclass in transistor.scheulers.brokers.queues which can be passed to thetasksparameter ofBaseWorkGroupManagerJust pass the appropriate connection string toExchangeQueueandBaseWorkGroupManagerand you can use either RabbitMQ or Redis as a message broker, thanks tokombu.in this case, theBaseWorkGroupManageralso acts as a AMQPconsumerwhich can receive messages from RabbitMQ message brokerworkers:aBaseWorkerobject encapsulates aSpiderobject like theSplashScraperorSplashCrawlerobjects, which has been customized by the end user to navigate and extract the targeted data from a structured web page.aBaseGroupobject can then be created, to encapsulate theBaseWorkerobject which contains theSpiderobject.The purpose of thisBaseGroupobject is to enable concurrency and scale by being able to spin up an arbitrarily large number ofBaseWorkerobjects, each assigned a different scrape task for execution.theBaseGroupobject can then receive tasks to execute, like individual book titles or electronic component part numbers to search, delegated by aWorkGroupManagerclass.eachBaseWorkerin theBaseGroupalso processes web request results, as they are returned from it’s wrappedSplashScraperobject.BaseWorkermethods include hooks for exporting data to mutiple formats like csv/xml or saving it to the db of your choice.eachBaseGroupshould be wrapped in aWorkGroupwhich is passed to theWorkGroupManager. Objects which theBaseWorkerwill use to process theSpiderafter it returns from the scrape should also be specified inWorkGroup, likeItems,ItemLoader, andExporter.managers:the overall purpose of theWorkGroupManagerobject is to provide yet more scale and concurrency through asynchronous I/O.TheWorkGroupManagercan spin up an arbitrarily large number ofWorkGroupobjects while assigning eachBaseWorker/Spiderin each of theWorkGroupobjects, individual scrape tasks.This design approach is most useful when you have a finite pipeline of scrape tasks which you want to search and compare the same terms, across multiple different websites, with each website targeted by oneWorkGroup.for example, we may have a list of 50 electronic component part numbers, which we want to search each part number in ten different regional websites. TheWorkGroupManagercan spin up aWorkGroupfor each of the 10 websites, assign 50 workers to eachWorkGroup, and send out 500BaseWorkerseach with 1 task to fill, concurrently.to further describe theWorkGroupManager, it is a middle-layer betweenStatefulBookandBaseGroup. It ingestsTaskTrackerobjects from theStatefulBookobject. It is also involved to switch states forTaskTrackerobjects, useful to track the task state like completed, in progress, or failed (this last detail is a work-in-progress).Persistenceexportersseetransistor/persistence/exportersexport data from aSpiderto various formats, includingcsv,xml,json,xml,pickle, andpretty printto afileobject.Object Storage, Search, and RetrievalTransistor can be used with the whichever database or persistence model you choose to implement. But, it will offer some open-source code in support of below:SQLAlchemywe useSQL Alchemyextensively and may include some contributed code as we find appropriate or useful to keep in the Transistor repository. At least, an example for reference will be included in theexamplesfolder.object-relational databaseusingPostgreSQLwithnewt.db.persist and store your custom python objects containing your web scraped data, directly in a PostgreSQL database, while also converting your python objects to JSON,automaticallyindexing them for super-quick searches, and making it available to be used from within your application or externally.leverage PostgreSQL’s strong JSON support as a document database while also enabling “ease of working with your data as ordinary objects in memory”.this is accomplished withnewt.dbwhich turnsPostgreSQLinto an object-relational database while leveraging PostgreSQL’s well integrated JSON support.newt.db is itself a wrapper built over the battle testedZODBpython object database andRelStoragewhich integrates ZODB with PostgreSQL.more on newt.db here[1]and here[2][1]Why Postgres Should Be Your Document Database (blog.jetbrains.com)[2]Newt DB, the amphibious database (newtdb.org).Database SetupTransistor maintainers prefer to use PostgreSQL with newt.db. Below is a quick setup walkthrough.After you have a valid PostgreSQL installation, you should install newt.db:pip install newt.dbAfter installation of newt.db you need to provide a URI connection string for newt.db to connect to PostgreSQL. An example setup might use two files for this, with a URI as shown
inexamples/books_to_scrape/settings.pyand a second file to setup newt.db as shown inexamples/books_to_scrape/persistence/newt_db.pyas shown below:examples/books_to_scrape/settings.pynot recreated here, check the source fileexamples/books_to_scrape/newt_db.py:importosimportnewt.dbfromexamples.books_to_scrape.settingsimportDevConfig,ProdConfig,TestConfigfromtransistor.utility.utilsimportget_debug_flagdefget_config():if'APPVEYOR'inos.environ:returnTestConfigreturnDevConfigifget_debug_flag()elseProdConfigCONFIG=get_config()ndb=newt.db.connection(CONFIG.NEWT_DB_URI)Next, we need to store our first two python objects in newt.db, which are:A list collection object, so we have a place to store our scrapes.An object to hold our list collection object, so that we can have a list of listsfromtransistor.persistence.newt_db.collectionsimportSpiderList,SpiderListsNow, from your python repl:fromtransistor.newt_dbimportndb>>>ndb.root.spiders=SpiderLists()# Assigning SpiderLists() is only required during initial setup. Or else, when/if you change the SpiderLists() object, for example, to provide more functionality to the class.>>>ndb.root.spiders.add('first-scrape',SpiderList())# You will add a new SpiderList() anytime you need a new list container. Like, every single scrape you save.  See ``process_exports`` method in ``examples/books_to_scrape/workgroup.py``.>>>ndb.commit()# you must explicitly commit() after each change to newt.db.At this point, you are ready-to-go with newt.db and PostgreSQL.Later, when you have a scraper object instance, such asBooksToScrapeScraper()which has finished it’s web scrape cycle, it will be stored in theSpiderList()namedfirst-scrapelike such:>>>ndb.root.spiders['first-scrape'].add(BooksToScrapeScraper(name="books.toscrape.com",book_title="Soumission"))More on StatefulBookPractical use requires multiple methods of input and output.StatefulBookprovides a method for reading an excel file
with one column of search terms,part numbersin the below example, which we would like to search and scrape data from multiple websites which sell such components:>>>fromtransistorimportStatefulBook>>>filepath='/path/to/your/file.xlsx'>>>trackers=['mousekey.cn','mousekey.com','digidog.com.cn','digidog.com']This will create four separate task trackers for each of the four websites to search with the part numbers:>>>book=StatefulBook(filepath,trackers,keywords="part_numbers")>>>book.to_do()Output:deque([<TaskTracker(name=mousekey.cn)>,<TaskTracker(name=mousekey.com)>,<TaskTracker(name=digidog.com.cn)>,<TaskTracker(name=digidog.com)>])So now, each website we intend to scrape, has it’s own task queue.  To work with an individual tracker and see what is in it’s individual to_do work queue:>>>fortrackerinbook.to_do():>>>iftracker.name=='mousekey.cn':>>>ms_tracker=tracker>>>print(ms_tracker)<TaskTracker(name=mousekey.cn)>>>>ms_tracker.to_do()deque(['050R30-76B','1050170001','12401598E4#2A','525591052','687710152002','ZL38063LDG1'])TestingThe easiest way to test your scraper logic is to download the webpage html and then pass in the html file with a test dict.
Below is an example:frompathlibimportPathdata_folder=Path("c:/Users/<your-username>/repos/<your-repo-name>/tests/scrapers/component/mousekey")file_to_open=data_folder/"mousekey.cn.html"f=open(file_to_open,encoding='utf-8')page=f.read()test_dict={"_test_true":True,"_test_page_text":page,"_test_status_code":200,"autostart":True}frommy_custom_scrapers.component.mousekeyimportMouseKeyScraperms=MouseKeyScraper(part_number='GRM1555C1H180JA01D',**test_dict)assertms.stock()=='17,090'assertms.pricing()=='{"1": "CNY ¥0.7888", "10": "CNY ¥0.25984", "100": "CNY ¥0.1102", '\'"500": "CNY ¥0.07888", "10,000": "CNY ¥0.03944"}'