Package: saas
Saas - Screenshot as a serviceInstallationRequirementsFUSEWhat is fuse?From theFUSE wikipedia pageFilesystem in Userspace (FUSE) is a software interface for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a "bridge" to the actual kernel interfaces.FUSE is used to mount a synthetic filesystem to read back the photos taken of the url given to saas. The user-space filesystem is dynamically filled with files and directories by saas. FUSE makes a good choice for this component since this can be easily integrated into almost any workflow, read more about this in theAPI section.ElasticsearchElasticsearchis used as a storage backend for saas. Read more about the storage in thestorage section.ImageMagickImageMagickis used for optimizing image files saved to disk. This is an optional dependency since it is only used when the--optimize-storageflag is used.Linux1. Install Elasticsearchusing dockersudodockerpulldocker.elastic.co/elasticsearch/elasticsearch:6.5.42. Install Firefox and Geckodriversudoapt-getinstallfirefox

wgethttps://github.com/mozilla/geckodriver/releases/download/v0.23.0/geckodriver-v0.23.0-linux64.tar.gz
tar-xvzfgeckodriver-v0.23.0-linux64.tar.gz
chmod+xgeckodriver
sudomvgeckodriver/usr/bin/3. Install ImageMagick (optional)sudoapt-getinstallimagemagick4. Install saas# Make sure you have Python 3.7 installed!python--version# Python 3.7.2pipinstallsaas

saas--version# saas 1.2.1macOS1. Install FUSE for macOSEither fromofficial website (recommended)or using homebrewbrewupdate
brewtaphomebrew/cask
brewcaskinstallosxfuse2. Install Elasticsearchbrewinstallelasticsearch3. Install Firefox and GeckodriverEither fromofficial websiteor using homebrewbrewcaskinstallfirefox4. Install Geckodriverbrewinstallgeckodriver5. Install Python 3.7brewinstallpython3
python3--version# Python 3.7.26. Install ImageMagick (optional)brewinstallimagemagick7. Install saas# Make sure you have Python 3.7 installed!python3--version# Python 3.7.2python3-mpipinstallsaas

saas--version# saas 1.2.1UsageGetting startedStart ElasticsearchEverytime you run saas you must make sure that there is an elasticsearch instance running and availible is availible for saas to connect to.If using dockersudodockerrun-d-p9200:9200-e"discovery.type=single-node"docker.elastic.co/elasticsearch/elasticsearch:6.5.4If binary exists in PATH# foregroundelasticsearch# or run it in the backgroundelasticsearch2>&1>elasticsearch.log&Taking a picture of a single URL#createinputfile$touchinput_urls#makemountpointforfilesystem$mkdirmount#startsaas#the--ignore-found-urlsoptionwilldisablethecrawlerbehaviour$saasinput_urlsmount--ignore-found-urlsmounting filesystem at: ./mountstarting 1 crawler threadsstarting 1 photographer threads#addurltoinputfile$echo"https://news.ycombinator.com/">>input_urls#thephotowillappearinsidethemountpoint$treemount/mount/└── news.ycombinator.com├── 2019011721│   └── index.png└── latest└── index.png3 directories, 2 filesUsing the crawlerThe crawler is a useful tool to find new urls to take pictuers of. It can be configured torun wildand crawl any domain it comes across, or stay at the domains that the urls in the input file belongs to.Stay at domainsUsing the--stay-at-domainflag the crawler will discard any domain that does not belong to the same domain as the page it was found at.$saasinput_urlsmount--stay-at-domain$echo"https://daringfireball.net/">>input_urls#afteraminuteorso$treemount/daringfireball.net/latest/mount/daringfireball.net/latest/├── 2006│   └── 06│       └── apple_open_source.png├── 2007│   └── 01│       └── enderle_leg_pulling.png├── 2008│   └── 04│       └── big_fan.png.rendering.saas├── 2017│   └── 07│       └── you_should_not_force_quit_apps.png├── 2019│   └── 01│       └── on_getting_started_with_regular_expressions.png├── index.png└── linked└── 2019└── 01└── 07└── samsung-itunes.png14 directories, 7 filesResetting the dataSince the mounted filesystem is a read-only filesystem simply removing the a photo from the filesystem is currently not possible.For now, at least, the best way to clear the data directory and the index is by using the--clear-data-dirand--clear-elasticsearchoptions#cannotmodifythemountedfilesystem$touchmount/footouch: mount/foo: Read-only file system#cleartheindexofurlsandphotometadata$saasinput_urlsmount--clear-elasticsearch#clearthephotofiles$saasinput_urlsmount--clear-data-dirRead more about storageSetting the viewport sizeThe camera viewport can be adjusted with the--viewport-widthand--viewport-heightoptions.By default the camera tries to take a full screen screenshot. This means that it figures out how tall a page is and resizes the camera height accordingly. Full screen screenshots take way longer time, especially on image-heavy sites.Full list of optionsusage: saas [-h] [--version] [--debug] [--refresh-rate] [--crawler-threads]
            [--photographer-threads] [--data-dir] [--clear-data-dir]
            [--elasticsearch-host] [--setup-elasticsearch]
            [--clear-elasticsearch] [--stay-at-domain] [--ignore-found-urls]
            [--viewport-width] [--viewport-height] [--viewport-max-height]
            [--optimize-storage] [--stop-if-idle]
            url_file mountpoint

Screenshot as a service

positional arguments:
  url_file              Path to input url file
  mountpoint            Where to mount filesystem via FUSE

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --debug               Display debugging information
  --refresh-rate        Refresh captures of urls every 'day', 'hour' or
                        'minute' (default: hour)
  --crawler-threads     Number of crawler threads, usually not neccessary with
                        more than one (default: 1)
  --photographer-threads
                        Number of photographer threads, beaware that
                        increasing too much won't neccessarily speed up
                        performance and hog the system (default: 1)
  --data-dir            Path to data directory (default: ~/.saas-data-dir)
  --clear-data-dir      Use flag to clear data directory on start
  --elasticsearch-host
                        Elasticsearch host (default: localhost:9200)
  --setup-elasticsearch
                        Use flag to create indices in elasticsearch
  --clear-elasticsearch
                        Use flag to clear elasticsearch on start, WARNING:
                        this will clear all indices found in elasticsearch
                        instance
  --stay-at-domain      Use flag to ignore urls from a different domain than
                        the one it was found at
  --ignore-found-urls   Use flag to ignore urls found on crawled urls
  --viewport-width      Width of camera viewport in pixels (default: 1920)
  --viewport-height     Height of camera viewport in pixels, if set to 0
                        camera will try to take a full height high quality
                        screenshot, which is way slower than fixed size
                        (default: 0)
  --viewport-max-height
                        Max height of camera viewport in pixels, if
                        --viewport-height is set this will be ignored
  --optimize-storage    Image files should be optimized to take up less
                        storage (takes longer time to render)
  --stop-if-idle        If greater than 0 saas will stop if it is idle for
                        more than the provided number of minutesStorageSaas uses two types of storages. A regular directory for storage of photo files, and elasticsearch for photo metadata and urls.ElasticsearchThe elastic search instance is configured by saas with three indicescrawledthis index holds urls that crawler have visited, the HTTP response code and any locks (meaning any photographer thread is taking a picture of that url)uncrawledthis index contains scraped urls from pages crawler have visitedphotosthis index contains photo metadata, file size, captured_at, filename etc.Data directoryWhen saas responds to a directory listing it only needs to query the elasticsearchphotosindex. Only when a read request is made, the actual file content is fetched from the data directory. The data directory holds the raw photo data with a unique id for each photo. Default path for this directory is~/.saas-data-dir$tree~/.saas-data-dir/├── 18│   └── 18dfe716-cdb2-4916-8154-6088d9bc6ee3.png├── 1c│   └── 1c1d0ee8-28f6-4b7c-b70f-8e800c58a3a6.png├── 29│   └── 29dd23f3-1791-46e6-8a83-25f5736a0894.png├── 50│   └── 50f13985-2cce-4464-942d-d9bbea165785.png├── 76│   └── 769933ce-2cde-4f30-a215-c26227850c8b.png├── 89│   ├── 8975f15c-7112-499c-97d5-44dd501b9b09.png│   └── 89ec9675-84f8-47fa-9589-8d39a8a34ea1.png├── ab│   └── ab5bbb0f-03cb-45ed-be1d-e257434a925c.png├── ca│   └── ca1551a2-8855-4d0d-869b-108b9b7122bf.png└── d7└── d79598a2-619f-4192-bb39-5e31642be800.pngBuildInstall saas by cloning it from source$gitclonehttps://github.com/nattvara/saas.git&&cdsaas$python3-mvenv./venv$source./venv/bin/activate$pythonsetup.pydevelopFirefox extensionsThe camera module uses selenium to render pages. To improve performance saas usesuBlock Originto block ads. To have greater access to more webpages saas usesI don't care about cookiesto bypass popups and GDPR consent forms. Many websites also employ the practice of paywalls for some of their content, however, many websites leave their site open to users coming from search engines and social media sites. Saas therefore has a small customfirefox extensionto rewrite all http requests made from firefox to include the headerReferer: https://google.com- this will allow access to a lot more content on the web.Updating uBlock OriginDownload the latest ublock.xpi fromgorhill/uBlock releasesand replace the version in theextensions/directory.Updating IDCACDownload and install the latest version using firefox fromhttps://www.i-dont-care-about-cookies.eu/. Locate the.xpifile inside Firefox's extensions directory, on macOS this is~/Library/Application Support/Firefox/Profiles/[profile]/extensions/. Copy the.xpifile to theextensions/directory.Referer HeaderMake zip archive of source fileszip-r-j-FSextensions/referer_header.xpiextensions/referer_header/*Run the testsuite$python-munittestdiscover-stestsRun the typechecker$mypysaasAPIThe main reason for using FUSE is that saas's apiis the filesystem. Everything that can interact with the filesystem can interact with saas. Almost every programming language ships with easy access to the filesystem, hence integration in any environment is as easy as reading and writing to the filesystem.For example exposing saas through a http interface could be as easy as starting a super simple node service like the following (should definitely be more thorough than this in production).consthttp=require('http')consturl=require('url')constfs=require('fs');constport=3000constrequestHandler=(request,response)=>{fs.appendFile('urls',url.parse(request.url,true).query.url+'\n',()=>{})response.end('')}constserver=http.createServer(requestHandler)server.listen(port,(err)=>{})This would allow for adding new urls to crawl by calling the service like the followingcurlhttp://localhost:3000/?url=https%3A%2F%2Fwww.wsj.com%2F
curlhttp://localhost:3000/?url=https%3A%2F%2Fwww.nytimes.com%2FStarting a simple python webserver could allow for traversing the saas filesystem# inside mounted filesystempython-mSimpleHTTPServer3001# so the following url# https://www.ft.com/content/180f3428-1923-11e9-b93e-f4351a53f1c3# if photographed, could be found atwgethttp://localhost:3001/www.ft.com/latest/content/180f3428-1923-11e9-b93e-f4351a53f1c3.pngThose are two out of a hundred ways to integrate/extend saas.Performance and ScalabilitySaas is designed to run over multiple machines. There can be virtually unlimited number of saas-nodes added to a single cluster, the only two things they need is a common elasticsearch instance or cluster to talk to, and a common data directory. Elasticsearch is well known for its scalability and the data directory could for instance be a network drive they share, Amazon EFS or any other way to share a drive between machines.Since all nodes in a cluster share the same index and data directory they can all read the images the cluster as a whole produces. Nodes can also join and leave the cluster freely without incurring any long time data loss.The biggest hit to performance are taking photos of image-heavy sites or using a large viewport size. Fixed viewport size is a good option for optimizing performance, there is virtually no upper limit to how large a website can be vertically. Screenshots of tabloid websites or sites with infinite-scroll can easily reach 25-50 MB in size.Checkout the guideMaximize saas throughputfor a thorough guide for how to deploy a large cluster of saas nodes on AWS and optimize performance.ExamplesSeeexamples/for some good examples for testing saas.Known issuesUnder some circumstances, a fatal crash for instance, the mounted filesystem might not unmount automatically. Also the filesystem will not be able to unmount if some other process is currently reading from the filesystem.If you encouter this, runumountpath/to/mounted_directoryLicenseMIT © Ludwig KristofferssonSeeLICENSE filefor more information