Package: webmixer
WebMixerA library for scraping urlsThe Basic ScraperAllwebmixer.scrapers.pagesandwebmixer.scrapers.tagsclasses inherit fromwebmixer.base.BasicScraper, which means they all have the following attributes and functions:Attributesdirectory (str): Directory to write files tocolor (str): Color for error messages (default: 'rgb(153, 97, 137)')locale (str): Language to use when writing error messages (default: 'en')Note: must be listed inwebmixer.messages.MESSAGESdefault_ext (str): Extension to default to for extracted filesFunctionscreate_tag(tag)Args:tag (str): tag name to create (e.g. 'p')Returns a BeautifulSoup tag
Example:image_tag = create_tag('img')get_filename(link, default_ext=None)Args:link (str): URL that has been scrapeddefault_ext (optional str): if the link doesn't have an extension, use this extension'Returns a filename (str) to use for extracted files
Example:video_filename = get_filename('<url>', default_ext='.mp4')mark_tag_to_skip(tag)Mark tag to skip during further scraping operations
Args:tag (str): tag to markExample:Process img tag here...

mark_tag_to_skip(img)write_url(link, url=None, default_ext=None, filename=None, directory=None)Downloads a url and writes it to a zip
Args:filepath (str): path to local filedirectory (str): directory to write to zipurl (optional str): URL used for handling relative URLsdefault_ext (optional str): if the link doesn't have an extension, use this extensionfilename (optional str): name for file to write to zipdirectory (optional str): directory to write file to zipReturns filepath within zip
Example:write_url('<link>', url='https://domain.com/', default_ext='.mp4', filename='video', directory='media') # 'media/video.mp4'write_contents(filename, contents, directory=None)Writes contents to the zip with a given filename
Args:filename (str): filename for contentscontents (bytes): contents to write to zipdirectory (str): directory to write to zipReturns filepath within zip
Example:write_contents('myfile.pdf', <pdf contents>, directory='docs')  # docs/myfile.pdfwrite_file(filepath, directory=None)Writes a local file to the zip
Args:filepath (str): path to local filedirectory (str): directory to write to zipReturns filepath within zip
Example:write_file('path/to/myfile.mp3', directory='music')  # music/myfile.mp3create_broken_link_message(link)Generates a tag with broken link message
Args:link (str): link to copy/pasteReturns a div tag with a link to copy/paste into browser
Example:iframe.replaceWith(create_broken_link_message('<url>'))
# iframe -> <div>copy link...</div>create_copy_link_message(link, partially_scrapable=False)Generates a tag with 'copy link into browser' message
Args:link (str): link to copy/pastepartially_scrapable (bool): link was mostly scraped, but doesn't include everything from original siteReturns a div tag with a link to copy/paste into browser
Example:iframe.replaceWith(create_copy_link_message('<url>'))
# iframe -> <div>copy link...</div>Exceptionswebmixer.exceptionscan be useful for handling errors from a variety of sources. If you are scraping a more specialized source, there may be some exceptions that are exclusive to that source. You can then raise the following exceptions to correctly manage that source:BrokenSourceExceptionUsed when the link is completely broken (e.g. site no longer exists)UnscrapableSourceExceptionUsed when the link is working, but cannot be supported on Kolibri (e.g. Flash content)For instance, thewebmixer.scrapers.pages.gdrive.GoogleDriveScrapermay throw aFileNotDownloadableErrorerror. In order to handle this correctly, it will raise anUnscrapableSourceExceptiontry:
	...
except FileNotDownloadableError as e:
	raise UnscrapableSourceException(e)Page ScrapersThere are several page scrapers that are available for use in scraping html pages. These will download urls to their respective file typesBuilt-in ScrapersHere is a list of the basic scraper classes, which are also listed underwebmixer.scrapers.pages.base.COMMON_SCRAPERS:WebVideoScraperPDFScraperEPubScraperImageScraperFlashScraperVideoScraperAudioScraperUsing Page ScrapersWhen you create a scraper object, you may specify the following:url (str): URL that tag can be found at (used to handle relative URLs)requiredzipper (optionalricecooker.utils.html_writer): Zip to write totriaged (optional [str]): List of already parsed URLsTo scrape the page, you may use any of the following writing options:to_zip: Writes a file to self.zipper, which is useful when scraping embedded sources from an html page
Args:filename (optional str): name of file to write to
Returns path to file from within zipHere are the default extensions for eachwebmixer.scrapers.pages.base.Scraper:ScraperExtensionHTMLPageScraper.htmlPDFScraper.pdfEPubScraper.epubAudioScraper.mp3VideoScraper.mp4WebVideoScraper.mp4ImageScraper.pngFlashScrapererrorFor example:from webmixer.scrapers.base import ImageScraper
image= <BeautifulSoup tag>
image['src'] = ImageScraper('<url>').to_zip()  # Sets 'src' to zipped image filepathto_tag: Writes file to zip and generates a tag based on what kind of scraper it is. This is useful when you are replacing iframes with native html elements
Args:filename (optional str): name of file to write to
Returns tagHere are the return tag types for eachwebmixer.scrapers.pages.base.Scraper:ScraperTagHTMLPageScraperNonePDFScraper<embed>EPubScraperNoneAudioScraper<audio>VideoScraper<video>WebVideoScraper<video>ImageScraper<img>FlashScrapererrorFor example:from webmixer.scrapers.base import PDFScraper
iframe= <BeautifulSoup tag>
iframe.replaceWith(PDFScraper('<url>').to_tag())  # Replaces iframe with <embed> tagto_file: Writes to a file. This is useful for downloading URLs as files to your local machine.
Args:filename (optional str): name of file to write todirectory (optional str): directory to write tooverwrite (bool): overwrite file if it exists
Returns a filepath to the downloaded fileto_fileuses thedownload_filemethod to write the file to awrite_to_pathHere are the return file types for eachwebmixer.scrapers.pages.base.Scraper:ScraperExtensionHTMLPageScraper.zip - generated byricecooker.utils.html_writerPDFScraper.pdfEPubScraper.epubAudioScraper.mp3VideoScraper.mp4WebVideoScraper.mp4ImageScrapererror - content kind not supportedFlashScrapererrorFor example:from webmixer.scrapers.base import HTMLPageScraper
new_html_zip_path = HTMLPageScraper('<url>').to_file() # Returns newly scraped html .zip fileCustom ScrapersGiven how diverse the internet is, you may need to implement your own scraper to handle individual sources. Youmustimplement atestclassmethod in order to use your scraper.If you would like to share a custom scraper, please feel free to open a pull request with a new file underwebmixer.scrapers.pagesAttributesAll scrapers have the following attributes:dl_directory (str): Directory to writeto_filedownloaded file to (default: 'downloads')directory (str): Directory to write files tocolor (str): Color for error messages (default: 'rgb(153, 97, 137)')locale (str): Language to use when writing error messages (default: 'en')Note: must be listed inwebmixer.messages.MESSAGESdefault_ext (str): Extension to default to for extracted fileskind (le_utils.constants.content_kind): Content kind to write towebmixer.scrapers.pages.base.HTMLPageScraperhas these additional attributes:partially_scrapable (bool): Not all content can be viewed from within Kolibri (default: False)scrape_subpages (bool): Determines whether to scrape any subpages within this page (default: True)main_area_selector (optional tuple): Main element selector to replace everything in body tagomit_list (optional list): list of selectors to remove from page contents (e.g. [('a', {'class': 'link'})])loadjs (bool): Determines whether to load js when loading the page (default: True)scrapers ([webmixer.scrapers.pages.Scraper]): List of additional scrapers to use on this pageextra_tags ([webmixer.scrapers.tags.Tag]): List of additional tags to scrapeFor example, the following code will remove links, scrape Wikipedia pages, and sets all images to 'myimg.png':from webmixer.scrpaers.tags import ImageTag
from webmixer.scrapers.pages.base import HTMLPageScraper
from webmixer.scrapers.pages.wikipedia import WikipediaScraper

class MyCustomTag(ImageTag):
	def process(self):
		self.tag['src'] = self.write_file('myimg.png')

class MyCustomScraper(HTMLPageScraper):
	omit_list = [('a',)]  		   # Remove links
	extra_tags = [MyCustomTag]     # Use MyCustomTag to set images to 'myimg.png'
	scrapers = [WikipediaScraper]  # Scrape any Wikipedia pages

	@classmethod                   # Required test classmethod
	def test(self, url):
		return 'my-domain.com' in urlFunctions@classmethod test(url): Required method to determine if this is the correct scraper for this URL
Args:url (str): url to test
Returns True if scraper is meant to scrape URL
Example:@classmethod
def test(self, url):
	return 'somedomain' in urlpreprocess(contents): Process contents before main scraping method
Args:
contents (BeautifulSoup): contents to preprocess
Example:# Delete the first image on the page before scraping all the images
def preprocess(self, contents):
	contents.find('img').decompose()postprocess(contents): Process contents after main scraping method
Args:
contents (BeautifulSoup): contents to postprocess
Example:# Append a link at the end of the <body> tag
def postprocess(self, contents):
	link = self.create_tag('a')
	link.string = 'New Link'
	contents.body.append(link)TagsThere are several tags that are available for use in scraping html pages. These will handle downloading any referenced files.Using TagsTo create a tag, you may specify the following:tag (BeautifulSoup.tag): tag to parserequiredurl (str): url that tag can be found at (used to handle relative URLs)requiredattribute (optional str): attribute to find link at (e.g. 'src' or 'data-src')scrape_subpages (optional bool): parse linked pages referenced by this tag (default: True)extra_scrapers (optional [webmixer.scrapers.base.BasicScrapers]): list of scrapers to try to scrape linked pagescolor (optional str): color for injected error messages (default: 'rgb(153, 97, 137)')To scrape the tag, use thescrapemethod. This will process the tag so that it can be usable from within an html zip. Here is a simple scraping example:from webmixer.scrapers.tags import ImageTag
image_tag = <BeautifulSoup.img tag>
image_scraper = ImageTag(image_tag, '<url>')
image_scraper.scrape()  # image_tag['src'] will point to downloaded image file in zipBuilt-in TagsHere is a list of the available tags, which are also listed underwebmixer.scrapers.tags.COMMON_TAGSImageTag (img)AudioTag (audio)VideoTag (video)EmbedTag (embed)LinkTag (a)Scrapes linked pages referenced by 'href' attributeIframeTag (iframe)Scrapes embedded pages referenced byon 'src' attributeStyleTag (style)Scrapes sheets referenced by 'href' attributeScriptTag (script)Scrapes scripts referenced by 'src' attributeCustom TagsDepending on the source you are trying to scrape, you may need more specific methods for scraping a page. To create a custom tag, you will need to subclasswebmixer.scrapers.tags.BasicScraperTagAttributesAll tags have the following attributes:selector (tuple): BeautifulSoup selector to find tag (e.g. ('a', {'class': 'link'}))default_ext (str): Extension to use if link doesn't have an extensiondirectory (str): Directory to write tag files toattributes (dict): Any attributes to assign to a tagdefault_attribute (str): Attribute that references files (default: 'src')scrape_subpages (bool): Determines whether to scrape any linked pages (default: True)extra_scrapers ([webmixer.scrapers.base.BasicScrapers]): List of additional scrapers to use for scraping linked pagescolor (str): Color for error messages (default: 'rgb(153, 97, 137)')locale (str): Language to use when writing error messages (default: 'en')Note: must be listed inwebmixer.messages.MESSAGESExample:from webmixer.scrapers.tags import BasicScraperTag

class MyVideoTag(BasicScraperTag):
	selector = ('video', {'class': 'video-class'})  # Select video.video-class
	directory = 'media'								# Files will be written to media folder
	attributes = {									# Videos will have width 100%
		'width': '100%'
	}Built-in functionsFor more custom scraping logic, you may also override the following methods:process(): Makes the tag usable from within an html zip by downloading any referenced files
Example:class MyVideoTag(BasicScraperTag):
	def process(self):
		# Scrape all of the <source> tags
	    for source in self.tag.find_all('source'):
	        BasicScraperTag(source, self.zipper, self.url).scrape()handle_error(): Determines how to handle cases where the link is broken
Example:class MyVideoTag(BasicScraperTag):
	def handle_error(self):
		self.tag.decompose()  # Just remove the element if it doesn't workhandle_unscrapable(): Determines how to handle cases where the link is not scrapable
Example:class MyVideoTag(BasicScraperTag):
	def handle_unscrapable(self):
		self.tag.replaceWith(self.create_copy_link_message(self.link))Helper Functionswebmixer.utils.guess_scraperIf you would like to determine which scraper to use based on a URL, you can use thewebmixer.utils.guess_scrapermethod. This will accept the following arguments:url (str): URL to scrapescrapers ([webmixer.scrapers.base.BasicScrapers]): list of other scrapers to test URL againstallow_defualt (optional bool): use generic default scraper in case nothing matches (default: False)You can also pass in additional arguments to scrapers withkwargsSo a simple usage ofguess_scrapermight be:from webmixer.utils import guess_scraper
scraper = guess_scraper('<url>', scrapers=[MyCustomScraper])=======
History0.0.0 (2019-07-30)First release on PyPI.