Package: recipe-scrapers
A simple web scraping tool for recipe sites.pipinstallrecipe-scrapersthen:fromrecipe_scrapersimportscrape_mescraper=scrape_me('https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/')# Q: What if the recipe site I want to extract information from is not listed below?# A: You can give it a try with the wild_mode option! If there is Schema/Recipe available it will work just fine.scraper=scrape_me('https://www.feastingathome.com/tomato-risotto/',wild_mode=True)scraper.host()scraper.title()scraper.total_time()scraper.image()scraper.ingredients()scraper.ingredient_groups()scraper.instructions()scraper.instructions_list()scraper.yields()scraper.to_json()scraper.links()scraper.nutrients()# not always availablescraper.canonical_url()# not always availablescraper.equipment()# not always availableYou also have an option to scrape html-like contentimportrequestsfromrecipe_scrapersimportscrape_htmlurl="https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/"html=requests.get(url).contentscraper=scrape_html(html=html,org_url=url)scraper.title()scraper.total_time()# etc...Notes:scraper.links()returns a list of dictionaries containing all of the <a> tag attributes. The attribute names are the dictionary keys.Scrapers available for:https://aberlehome.com/https://claudia.abril.com.br/https://abuelascounter.com/https://www.acouplecooks.comhttps://addapinch.com/http://www.afghankitchenrecipes.com/https://akispetretzikis.com/https://ah.nl/https://allrecipes.com/https://alltommat.se/https://altonbrown.com/https://amazingribs.com/https://ambitiouskitchen.com/https://archanaskitchen.com/https://www.argiro.gr/https://www.arla.se/https://www.atelierdeschefs.fr/https://averiecooks.com/https://barefootcontessa.com/https://baking-sense.com/https://bakingmischief.com/https://bbc.com/https://bbc.co.uk/https://bbcgoodfood.com/https://bettybossi.ch/https://bettycrocker.com/https://biancazapatka.com/https://bigoven.com/https://blueapron.com/https://bluejeanchef.com/https://bonappetit.com/https://www.bodybuilding.com/https://bongeats.com/https://bowlofdelicious.com/https://briceletbaklava.ch/https://budgetbytes.com/https://carlsbadcravings.com/https://castironketo.net/https://cdkitchen.com/https://chefkoch.de/https://www.chefnini.com/https://chefsavvy.com/https://closetcooking.com/https://comidinhasdochef.com/https://cookeatshare.com/https://cookieandkate.com/https://cookingcircle.com/https://cookinglight.com/https://cookpad.com/https://cookstr.com/https://cook-talk.com/https://www.coop.se/https://copykat.com/https://www.costco.com/https://countryliving.com/https://creativecanning.com/https://cucchiaio.it/https://cuisineaz.com/https://cybercook.com.br/https://www.davidlebovitz.com/https://delish.com/https://domesticate-me.com/https://downshiftology.com/https://www.dr.dk/https://www.eatingbirdfood.com/https://www.eatingwell.com/https://www.eatliverun.com/https://eatsmarter.com/https://eatsmarter.de/https://www.eatwell101.comhttps://eatwhattonight.com/https://elavegan.com/https://emmikochteinfach.de/https://ethanchlebowski.com/https://epicurious.com/https://www.errenskitchen.com/https://recipes.farmhousedelivery.com/https://www.farmhouseonboone.com/https://www.fattoincasadabenedetta.it/https://fifteenspatulas.com/https://finedininglovers.com/https://fitmencook.com/https://fitslowcookerqueen.comhttps://food.com/https://food52.com/https://foodandwine.com/https://foodnetwork.com/https://foodrepublic.com/https://www.forksoverknives.com/https://forktospoon.com/https://fredriksfika.allas.se/https://www.750g.comhttps://www.gesund-aktiv.com/https://giallozafferano.it/https://gimmesomeoven.com/https://godt.no/https://goodfooddiscoveries.com/https://goodhousekeeping.com/https://recietas.globo.com/https://gonnawantseconds.com/https://gousto.co.uk/https://www.grandfrais.com/https://greatbritishchefs.com/https://grimgrains.com/http://www.grouprecipes.com/https://halfbakedharvest.com/https://handletheheat.com/https://www.hassanchef.com/https://headbangerskitchen.com/https://heatherchristo.com/https://www.heb.com/https://hellofresh.com/https://hellofresh.co.uk/https://www.hellofresh.de/https://www.hellofresh.fr/https://www.hellofresh.nl/https://www.hellofresh.ie/https://www.hersheyland.com/https://www.homechef.com/https://hostthetoast.com/https://www.ica.se/https://receitas.ig.com.br/https://www.im-worthy.com/https://indianhealthyrecipes.comhttps://www.innit.com/https://insanelygoodrecipes.comhttps://inspiralized.com/https://izzycooking.com/https://jamieoliver.com/https://jimcooksfoodgood.com/https://joshuaweissman.com/https://joyfoodsunshine.com/https://juliegoodwin.com.au/https://justataste.com/https://justbento.com/https://www.justonecookbook.com/https://kennymcgovern.com/https://www.kingarthurbaking.comhttps://www.kitchenstories.com/https://kochbar.de/https://kochbucher.com/http://koket.se/https://www.kptncook.com/https://kuchnia-domowa.pl/https://www.kwestiasmaku.com/https://www.latelierderoxane.comhttps://leanandgreenrecipes.nethttps://lecremedelacrumb.com/https://www.lecker.dehttps://lekkerensimpel.comhttps://lifestyleofafoodie.comhttps://littlespicejar.com/http://livelytable.com/https://lovingitvegan.com/https://www.maangchi.comhttps://madensverden.dk/https://www.madewithlau.com/https://madsvin.com/https://marleyspoon.com.au/https://marleyspoon.com/https://marleyspoon.de/https://marleyspoon.at/https://marleyspoon.be/https://marleyspoon.nl/https://marleyspoon.se/https://marmiton.org/https://www.marthastewart.com/https://matprat.no/https://meljoulwan.com/https://www.melskitchencafe.com/http://mindmegette.hu/https://minimalistbaker.com/https://ministryofcurry.com/https://misya.info/https://www.mob.co.uk/https://momswithcrockpots.com/https://monsieur-cuisine.com/http://motherthyme.com/https://www.mundodereceitasbimby.com.pt/https://mybakingaddiction.com/https://mykitchen101.com/https://mykitchen101en.com/https://www.myplate.gov/https://myrecipes.com/https://healthyeating.nhlbi.nih.gov/https://nibbledish.com/https://www.nhs.uk/healthier-families/https://www.nosalty.hu/https://nourishedbynutrition.com/https://www.nrk.no/https://www.number-2-pencil.com/https://nutritionbynathalie.com/bloghttps://nutritionfacts.org/https://cooking.nytimes.com/https://ohsheglows.com/https://omnivorescookbook.comhttps://www.onceuponachef.comhttps://owen-han.com/https://101cookbooks.com/https://www.paleorunningmomma.com/https://www.panelinha.com.br/https://paninihappy.com/https://www.persnicketyplates.com/https://pinchofyum.com/https://www.pickuplimes.com/https://www.pingodoce.pt/https://pinkowlkitchen.com/https://www.platingpixels.com/https://plowingthroughlife.com/https://popsugar.com/https://practicalselfreliance.com/https://pressureluckcooking.com/https://www.primaledgehealth.com/https://www.projectgezond.nl/https://przepisy.pl/https://purelypope.com/https://purplecarrot.com/https://rachlmansfield.com/https://rainbowplantlife.com/https://realfood.tesco.com/https://realsimple.com/https://recept.se/https://www.receitasnestle.com.brhttps://reciperunner.com/https://recipetineats.com/https://redhousespice.com/https://reishunger.de/https://rezeptwelt.de/https://ricetta.it/https://www.ricetteperbimby.it/https://rosannapansino.comhttps://rutgerbakt.nl/https://www.saboresajinomoto.com.br/https://sallysbakingaddiction.comhttps://sallys-blog.dehttps://saltpepperskillet.com/https://www.saveur.com/https://seriouseats.com/https://simple-veganista.com/https://simplyquinoa.com/https://simplyrecipes.com/https://simplywhisked.com/https://simply-cookit.com/https://skinnytaste.com/https://sobors.hu/https://www.southerncastiron.com/https://southernliving.com/https://spendwithpennies.com/https://www.springlane.dehttps://www.staysnatched.com/https://steamykitchen.com/https://streetkitchen.hu/https://www.strongrfastr.comhttps://sunbasket.com/https://sundpaabudget.dk/https://www.sunset.com/https://sweetcsdesigns.com/https://sweetpeasandsaffron.com/https://www.taste.com.au/https://tasteofhome.comhttps://tastesbetterfromscratch.comhttps://tastesoflizzyt.comhttps://tasty.cohttps://tastykitchen.com/https://theclevercarrot.com/https://theexpertguides.com/https://thehappyfoodie.co.uk/https://www.thekitchenmagpie.com/https://thekitchencommunity.org/https://thekitchn.com/https://www.themagicalslowcooker.com/https://themodernproper.com/https://www.thepalatablelife.comhttps://thepioneerwoman.com/https://therecipecritic.com/https://thespruceeats.com/https://thevintagemixer.com/https://thewoksoflife.com/https://thinlicious.com/https://timesofindia.com/https://tine.no/https://tidymom.nethttps://tudogostoso.com.br/https://twopeasandtheirpod.com/https://uitpaulineskeuken.nl/https://usapears.org/https://www.valdemarsro.dk/https://vanillaandbean.com/https://www.vegetarbloggen.no/https://vegolosi.it/https://vegrecipesofindia.com/https://www.waitrose.com/https://watchwhatueat.com/https://wearenotmartha.com/https://www.weightwatchers.com/(*)https://www.wellplated.com/https://whatsgabycooking.com/https://www.wholefoodsmarket.com/https://www.wholefoodsmarket.co.uk/https://www.williams-sonoma.com/https://woop.co.nz/https://woolworths.com.au/shop/recipeshttps://en.wikibooks.org/https://yemek.com/https://yummly.com/(*)https://www.zaubertopf.dehttps://zeit.de/ (wochenmarkt)https://zenbelly.com/(*) offline saved files onlyContributeIf you spot a design change (or something else) that makes the scraper unable to work for a given site - please fire an issue asap.If you are programmer PRs with fixes are warmly welcomed and acknowledged with a virtual beer. You can find documentation on how to develop scrapershere.If you want a scraper for a new site addedOpen anIssueproviding us the site name, as well as a recipe link from it.You are a developer and want to code the scraper on your own:IfSchema is availableon the site -you can go like this.Otherwise, scrape the HTML -like thisGenerating a new scraper class:pythongenerate.py<ClassName><URL>ClassName: The name of the new scraper class.URL: The URL of an example recipe from the target site. The content will be stored intest_datato be used with the test class.You can find a more detailed guidehere.For Devs / ContributeAssuming you have>=python3.8installed, navigate to the directory where you want this project to live in and drop these linesgitclonegit@github.com:hhursev/recipe-scrapers.git&&cdrecipe-scrapers&&python-mvenv.venv&&source.venv/bin/activate&&python-mpipinstall--upgradepip&&pipinstall-rrequirements-dev.txt&&pipinstallpre-commit&&pre-commitinstall&&python-munittestIn case you want to run a single unittest for a newly developed scraperpython-munittest-k<test_file_name>FAQHow do I know if a website has a Recipe Schema?Run in python shell:fromrecipe_scrapersimportscrape_mescraper=scrape_me('<url of a recipe from the site>',wild_mode=True)# if no error is raised - there's schema available:scraper.title()scraper.instructions()# etc.NetiquetteIf you’re using this library to collect large numbers of recipes from the web, please use the software responsibly and try to avoid creating high volumes of network traffic.Python’s standard library provides arobots.txtparserthat may be helpful to automatically follow common instructions specified by websites for web crawlers.Another parser option – particularly if you find that many web requests fromurllib.robotparserare blocked – is therobotexclusionrulesparserlibrary.Special thanks to:All thecontributors that helped improvingthe package. You are awesome!Extra:You want to gather recipes data?You have an idea you want to implement?Check outour “Share a project” wall- it may save you time and spark ideas!