Package: cardscraper
cardscraperWebscraping tool for generating Anki packages.InstallationFromPyPI:pipinstallcardscraperFrom git:pipinstallgit+https://github.com/sakhezech/cardscraperUsagecardscraper ...orpython -m cardscraper ...Generate a skeleton input file:cardscraperinitfilename.yamlEdit it with your favorite text editor:nvimfilename.yamlGenerate the package:cardscrapergenfilename.yamlFor more info usecardscraper -h.Input filesYou can generate a skeleton input file by usingcardscraper init filename.yaml.Here is a big self-explaining input file example:# here you can specify which function to use for each step# (every one defaults to 'default')meta:# controls package details and package dumpingpackage:default# controls deck creationdeck:default# controls model creationmodel:default# controls scraping and note creationscraping:default# anki package infopackage:# package namename:package_name# output folder (defaults to '.')output:./out/# media folder (defaults to null)# the directory will be walked recursively# every pattern matched file will be added to the package as mediamedia:./media/# pattern to match files against for media (defaults to **/*.*)pattern:"**/*.png"# anki deck infodeck:# deck namename:Deck# deck id# don't forget to make this value uniqueid:987# anki model infomodel:# model namename:Model# model id# don't forget to make this value uniqueid:321# card styling (defaults to '')css:|.question, .answer {text-align: center;}.question {font-size: 5rem;font-weight: 700;}.answer {font-size: 3rem;}# list of cardstemplates:# card name-name:Front# front sideqfmt:|<div class='question'>{{Question}}</div># back sideafmt:|{{FrontSide}}<hr id='answer'><div class='answer'>{{Answer}}</div># same here-name:Backqfmt:|<div class='question'>{{Answer}}</div>afmt:|{{FrontSide}}<hr id='answer'><div class='answer'>{{Question}}</div># scraping infoscraping:# list of urls to scrapeurls:-https://www.scrapethissite.com/pages/simple/# you can set your own custom user agent (defaults to null)agent:Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0# list of queries# each query selects an html element and lets you use its text in the templates# each child query runs inside the parent onequeries:# query name which you can use in the templates like {{Country}}-name:Country# css selectorquery:.country# you can select something specific from the query by providing a regex# this is a python regex with re.DOTALL enabled i.e. '.' captures '\n'# uses the first captured group# (defaults to null)regex:null# if true: we select every instance and iterate over them# if false: we only select the first one# basically it's querySelector() vs querySelectorAll()# (defaults to false)many:truechildren:-name:Questionquery:.country-infomany:falseregex:(Area .*)$children:null-name:Answerquery:.country-namemany:falseregex:nullchildren:nullUsage in codeIt is possible to use cardscraper programmatically, but it is created to be used as a CLI application.importyamlfromcardscraperimport(Config,generate_anki_package,select_function_by_step_and_name,write_package,)fromgenankiimportModel,Noteif__name__=='__main__':withopen('/path/to/config.yaml','r')asf:config:Config=yaml.load(f,yaml.Loader)# or you can make a config manuallyget_model=select_function_by_step_and_name('model','default')get_deck=select_function_by_step_and_name('deck','default')get_package=select_function_by_step_and_name('package','default')defget_notes(config:Config,model:Model)->list[Note]:notes=[]...returnnotespackage,path=generate_anki_package(config,get_model,get_notes,get_deck,get_package)write_package(package,path)Plugin systemA plugin system is present in cardscraper. To expose your functions to cardscraper expose them in an entry point namedcardscraper.STEPNAME.This is how the default functions are exposed:[project.entry-points.'cardscraper.model']default='cardscraper.default:get_model'[project.entry-points.'cardscraper.scraping']default='cardscraper.default:get_notes'[project.entry-points.'cardscraper.deck']default='cardscraper.default:get_deck'[project.entry-points.'cardscraper.package']default='cardscraper.default:get_package'