Package: webpreview
webpreviewFor a given URL,webpreviewextracts itstitle,description, andimage urlusingOpen Graph,Twitter Card, orSchemameta tags, or, as an alternative, parses it as a generic webpage.InstallationpipinstallwebpreviewUsageUse the genericwebpreviewmethod (added inv1.7.0) to parse the page independent of its nature.
This method fetches a page and tries to extracts atitle, description, and a preview imagefrom it.It first attempts to parse the values fromOpen Graphproperties, then it falls back toTwitter Cardformat, and then toSchema. If none of these methods succeed in extracting all
three properties, then the web page's content is parsed using a generic HTML parser.>>>fromwebpreviewimportwebpreview>>>p=webpreview("https://en.wikipedia.org/wiki/Enrico_Fermi")>>>p.title'Enrico Fermi - Wikipedia'>>>p.description'Italian-American physicist (1901–1954)'>>>p.image'https://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Enrico_Fermi_1943-49.jpg/1200px-Enrico_Fermi_1943-49.jpg'# Access the parsed fields both as attributes and items>>>p["url"]==p.urlTrue# Check if all three of the title, description, and image are in the parsing result>>>p.is_complete()True# Provide page content from somewhere else>>>content="""<html><head><title>The Dormouse's story</title><meta property="og:description" content="A Mad Tea-Party story" /></head><body><p class="title"><b>The Dormouse's story</b></p><a href="http://example.com/elsie" class="sister" id="link1">Elsie</a></body></html>"""# The the function's invocation won't make any external calls,# only relying on the supplied content, unlike the example above>>>webpreview("aa.com",content=content)WebPreview(url="http://aa.com",title="The Dormouse's story",description="A Mad Tea-Party story")Using the command lineWhenwebpreviewis installed viapip, then the accompanying command-line tool is
installed alongside.$webpreviewhttps://en.wikipedia.org/wiki/Enrico_Fermi
title:EnricoFermi-Wikipedia
description:Italian-Americanphysicist(1901–1954)image:https://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Enrico_Fermi_1943-49.jpg/1200px-Enrico_Fermi_1943-49.jpg

$webpreviewhttps://github.com/--absolute-url
title:GitHub:Wheretheworldbuildssoftware
description:GitHubiswhereover83milliondevelopersshapethefutureofsoftware,together.
image:https://github.githubassets.com/images/modules/site/social-cards/github-social.pngUsing compatibility APIBeforev1.7.0the package mainly exposed a different set of the API methods.
All of them are supported and may continue to be used.# WARNING:# The API below is left for BACKWARD COMPATIBILITY ONLY.fromwebpreviewimportweb_previewtitle,description,image=web_preview("aurl.com")# specifing timeout which gets passed to requests.get()title,description,image=web_preview("a_slow_url.com",timeout=1000)# passing headersheaders={'User-Agent':'Mozilla/5.0'}title,description,image=web_preview("a_slow_url.com",headers=headers)# pass html content thus avoiding making http call again to fetch content.content="""<html><head><title>Dummy HTML</title></head></html>"""title,description,image=web_preview("aurl.com",content=content)# specifing the parser# by default webpreview uses 'html.parser'title,description,image=web_preview("aurl.com",content=content,parser='lxml')Run with DockerThe docker image can be built and ran similarly to the command line.
The default entry point is thewebpreviewcommand-line function.$dockerbuild-twebpreview.
$dockerrun-it--rmwebpreview"https://en.m.wikipedia.org/wiki/Enrico_Fermi"title:EnricoFermi-Wikipedia
description:EnricoFermi(Italian:[enˈriːkoˈfermi];29September1901–28November1954)wasanItalian(laternaturalizedAmerican)physicistandthecreatoroftheworld'sfirstnuclearreactor,theChicagoPile-1.Hehasbeencalledthe"architect of the nuclear age"[1]andthe"architect of the atomic bomb".
image:https://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Enrico_Fermi_1943-49.jpg/1200px-Enrico_Fermi_1943-49.jpgNote: built docker image weighs around 210MB.Testing# Execute the testspoetryrunpytestwebpreview# OR execute until the first failed testpoetryrunpytestwebpreview-xSetting up development environment# Install a correct minimal supported version of pythonpyenvinstall3.7.13# Create a virtual environment# By default, the project already contains a .python-version file that points# to 3.7.13.python-mvenv.venv# Install dependencies# Poetry will automatically install them into the local .venvpoetryinstall# If you have errors likes this:ERROR:Cannotexecute`setup.py`sincesetuptoolsisnotavailableinthebuildenvironment.# Then do this:.venv/bin/pipinstall--upgradesetuptools