Package: pickmeup
PickMeUp - Pick up failed list processing without re-processingPickMeUpallows to pick up a failed processing of elements.elements=[1,2,3,4,5,6,7,8]withPickMeUp(elements,name="element_processing")asl:foreinl:result=process(e)save(result)Ifprocessfails for elements dividable by 4, the script crashes. You now have to fixprocessand re-run the script.
Normally, you would re-process 1, 2, and 3. However, you already have results for those! Andprocessmight be expensive.PickMeUpwill know that the processing failed at 4, and will only iterate over 4, 5, 6, 7, and 8, saving you from re-processing.Without PickMeUpProcess 1,2,3FixprocessProcess 1,2,3,4,5,6,7,8With PickMeUpProcess 1,2,3FixprocessProcess 4,5,6,7,8Installationpip install pickmeupUsageCreate a context using your list and a name, and use the context element as your list:withPickMeUp(yourList,"some_unique_name")asl:foreinl:# Do stuff with eExample Use-Case: ScrapingImagine you want to crawl content from a website.
You have a list of 50 content pages, and you want to extract thesrcof one specific image from each page.
You wrote the extractor (identifying the important image, extracting thesrc) by looking at the first 2 pages.
It seems that the important image can be identified with theid=important-image:importrequestsfrombs4importBeautifulSoupdefscrape(url:str)->str:"""Returns the html of the given `url`"""resp=requests.get(url)returnresp.contentdefextract_important_image_src(html:str)->str:"""Finds the important image in the given `html` andreturns its `src` attribute"""soup=BeautifulSoup(html)returnsoup.find('img',{'id':'important-image'})['src']URLS=["example.com/content/1","example.com/content/2",...]forurlinURLS:html=scrape(url)src=extract_important_image_src(html)print(f"{src}extracted!")However, it just so happens that half of the content pages follow an old design, and those pages mark their image withid=important.
Your extractor breaks, because there is no element withid=important-imageand the subscript['src']fails:>"example.com/content/1 extracted!">...
>"example.com/content/25 extracted!">TypeError:'NoneType'objectisnotsubscriptableSo you enhanceextract_important_image_srcto handle the old design. But now you have to parse the first 25 pages again!
Or skip them manually. Instead, if you usedPickMeUp:URLS=["example.com/content/1","example.com/content/2",...]withPickMeUp(URLS,name="url_processing")asl:forurlinl:html=scrape(url)src=extract_important_image_src(html)print(f"{src}extracted!")You could just updateextract_important_image_srcand re-run your script.
It will pick up the scraping at the element that failed, and proceed as if nothing happened.
If there is another design change somewhere, the processing will fail again. But again, you can fix the issue and re-run the script,
without re-processing all the urls that worked before!CaveatsPickMeUpcreates state for your processing by dumping the remaining elements to disk. This creates a few caveats:If your elements are really large objects, this will take much disk space.If you are using a generator, and the elements are expensive to create,PickMeUpwill take the time to create them all and dump them to disk.Changing the list elements between script runs without clearing state creates undefined behavior!List elements have to be serializable!