Package: lapdog
LapdogA relaxed wrapper for dalmatian and FISSPrerequisitesLapdog requires MacOS or Linux. Windows is not officially supportedLapdog requires Python >= 3.3Lapdog requires the Google Cloud SDK, which can be installedhere.Yourgcloud --versionshould be at least252.0.0. If it is not, please rungcloud components updateLapdog uses your Gcloud application-default credentialsUsegcloud auth application-default loginandgcloud config set account {account}before running LapdogLapdog requires the Alpha and Beta suite for GcloudAlpha and Beta suite can be installed withgcloud components install alpha beta(Optional) The Lapdog User Interface requires node and npm, which can be installed using your system's package managerMacOS:brew install node npmLinux: Install through system package manager orfrom sourceLapdog requiresnode --version>= 10.15.1Lapdog requiresnpm --version>= 6.4.1InstallingInstall lapdog via pip:pip install lapdogIf you already have lapdog installed, you can upgrade it withpip install --upgrade lapdogIf you encounter an error with PyYAML see thePyYaml Notebelow(Optional) Enable the Lapdog User Interface:The UI runs locally by default. If you are installing Lapdog on a server, you'll
need to set up an SSH tunnel for ports 4200 and 4201Installnodeandnpmif you don't already have them installedIf you're on Mac OS, runbrew install node npmRunlapdog ui --install. This may take a whileUsagelapdogmay be imported within python as a drop-in replacement fordalmatianlapdog presents a superset of features available in dalmatianWorkspaceManagers in lapdog cache data when communicating with Firecloud.
If Firecloud experiences an intermittent failure, theWorkspaceManagermay be
able to continue running in offline mode. CallingWorkspaceManager.sync()will
reconnect to Firecloud, pushing out any data updates that were queued while in offline modeWorkspaceManagers in lapdog present the execution api viaWorkspaceManager.execute().
Executions differ from submissions in that they run directly on Google and results are
uploaded back to Firecloud afterwardslapdogmay be used as a command line tool.The tool provides the necessary functions to create a workspace, fill it with data,
import or upload methods and configurations, and submit jobs (or execute them directly)Runlapdog --helpto get the list of available commandslapdogmay be used via an interactive user interface which serves to run and
monitor lapdog executionsRunlapdog uito launch the user interfaceJob ExecutionLapdog executes jobs through dedicated Google Projects ("Engines") for each FireCloud Namespace.
A Lapdog Engine can only be initialized for a given Namespace by a billing account admin.
To initialize a new Engine, contact your Namespace admin and ask them to runlapdog initialize-project.After an Engine is initialized, you will have to register with it:The Lapdog User Interface will automatically register you when you load a workspace
in a namespace that you're not registered toThe Lapdog python module supports manual registrationWhen you create aWorkspaceManagerin an unregistered Namespace, you will get a warningYou can also check your registration status by checking the value ofWorkspaceManager.gateway.registeredYou can then register by usingWorkspaceManager.gateway.register()If registration fails due to any FireCloud errors, simply wait a few minutes
then try callingregister()againThe Lapdog CLI does not support registration. You can register through the UI or
python moduleWorkspace PermissionsIn the UI, at the bottom of every page, you will find afirecloud.orgemail.
This is a proxy group email which contains you, and all your service accounts.
To allow the Lapdog Engine to run jobs, that proxy group email must be granted
WRITE access to FireCloud workspaces where jobs will run. You may grant the group
READ access to workspaces where data will be read from, but jobs cannot execute
in workspaces without WRITE permissions. The proxy group email can be found by
callinglapdog.cloud.proxy_group_for_user(YOUR_EMAIL).NOTE:Due to a bug in FireCloud, permissions will not be granted if the group
was already granted access to a workspace before you registered to that namespace's
Lapdog Engine. If your proxy email definitely was granted access to a workspace,
but your jobs are still failing with permissions errors, try removing access and then
re-granting it. You can see FireCloud's response to this bug reporthereRoadmapSee themilestones pageto
read the development roadmap.Pro/Con with FirecloudProsEach submission has a dedicated Cromwell instance. Your jobs will never queue, unless you hit a Google usage quotaLapdog supports Requester Pays buckets and GPUsWorkspace cache: Lapdog caches most data received from Firecloud.In the event of a Firecloud error, Lapdog will attempt to keep running by using it's cached data. Any data updates will by pushed back to Firecloud when the workspace is syncedData caches: The Lapdog API caches data sent to the UI and read from GoogleThese caches greatly improve UI performance by storing results whenever possibleStreamlined UI: The Lapdog UI was built with efficiency in mindQuality of life features:Save time updating methods. SetmethodRepoMethod.methodVersionto "latest" and let Lapdog figure out what the snapshot ID isEasy data uploads. Callprepare_entity_dfon a DataFrame before uploading to Firecloud. Any local filepaths will be uploaded to the workspace's bucket in the background and a new DataFrame will be returned containing the newgs://pathsAutomatic reference uploads. When you callupdate_attributes, any values which refer to local filepaths will be uploaded in the background (just likeprepare_entity_df).update_attributesnow returns a dictionary containing the attributes exactly as uploadedConsYou pay an additional 5c/hour fee for each submission to run the Cromwell serverSubmission results must be manually uploaded to Firecloud by clicking theUpload Resultsbutton in the UI.There are small overhead costs billed to the Lapdog Engine for operation. These costs
are for calls to the API and for storage of metadata, both of which should be very cheapPyYAML NoteOften, when installing Lapdog, the installation fails when upgrading PyYAML because
it is unable to uninstall the current PyYAML version.This is because some older versions of PyYAML were distributed throughdistutilswhich prevents packages from being uninstalled. New versions of PyYAML are distributed
throughsetuptoolswhich can be successfully uninstalled.To resolve this issue, navigate to your Python site-packages directory. The site-packages
directory usually ends with.../lib/python{version}/site-packages/. The exact location
depends on your platform, environment manager, and python configuration.For anaconda, this path is within anaconda's installation folder:/{path to anaconda}/envs/{environment}/lib/python{version}/site-packagesFor virtualenvs, this path is within the environment's installation folder:/{path to environment}/lib/python{version}/site-packagesFor system python on Unix systems:/usr/local/lib/python{version}/site-packagesAn easy way to locate this folder is to open python and then:importyamlyamlThis expression will evaluate and print the path to theyamlmodule, which will be
within your site-packages directory.After locating the site-packages directory, you must remove the following two directories:.../site-packages/yaml/.../site-packages/PyYAML*.egg-infoOnce removed, you can try the installation command for lapdog again.