Package: rhg-compute-tools
Tools for using compute.rhg.com and compute.impactlab.orgFree software: MIT licenseDocumentation:https://rhg-compute-tools.readthedocs.io.Installationpip:pipinstallrhg_compute_toolsFeaturesKubernetes toolseasily spin up a preconfigured cluster withget_cluster(), or flavors withget_micro_cluster(),get_standard_cluster(),get_big_cluster(), orget_giant_cluster().>>>importrhg_compute_tools.kubernetesasrhgk>>>cluster,client=rhgk.get_cluster()Google cloud storage utilitiesUtilities for managing google cloud storage directories in parallel from the command line or via a python API>>>importrhg_compute_tools.gcsasgcs>>>gcs.sync_gcs('my_data_dir','gs://my-bucket/my_data_dir')HistoryCurrent version (unreleased)Addutils.get_repo_stateandxarray.document_datasetfunctionsDrop explicit testing of dask.gateway rpy2 functionalityBugfix in sphinx docsv1.2.1Bug fixes:
* raise error on gsutil nonzero status inrhg_compute_tools.gcs.cp(PR #105)v1.2New features:
* Adds google storage directory marker utilities and rctools gcs mkdirs command line appv1.1.4Adddask_kwargsto therhg_compute_tools.xarrayfunctionsv1.1.3Addretry_with_timeouttorhg_compute_tools.utils.pyv1.1.2Dropmatplotlib.font_manager._rebuild()call indesign.__init__- no longer supportedv1.1.1Refactordatasets_from_delayedto speed upv1.1Addgcs.lsfunctionv1.0.1Fixtagkwarg inget_clusterv1.0.0Make the gsutil API consistent, so that we havecp,syncandrm, each of which
accept the same args and kwargsSwapbumpversionforsetuptools_scmto handle versioningCast coordinates to dict before gathering inrhg_compute_tools.xarray.dataarrays_from_delayedandrhg_compute_tools.xarray.datasets_from_delayed. This avoids a mysterious memory explosion on the local machine. Also addnamein the metadata used by those functions so that the name of each dataarray or Variable is preserved.Usedask-gatewaywhen available when creating a cluster inrhg_compute_tools.kubernetes. Add some tests using a local gateway cluster. TODO: More tests.Addtagkwarg torhg_compute_tools.kuberentes.get_clusterfunction (PR #87)v0.2.2?v0.2.1Add remote scheduler deployment (part of dask_kubernetes 0.10)Remove extraneousGCSFUSE_TOKENSenv var no longer used in new worker imagesSet library thread limits based on how many cpus are available for a single dask threadChange formatting of the extraenv_itemspassed toget_clusterto be a list rather than a list of dict-like name/value pairsv0.2.0Add CLI tools . Seerctools gcs repdirstruc--helpto startAdd new functionrhg_compute_tools.gcs.replicate_directory_structure_on_gcsto copy directory trees into GCS. Users can authenticate with cred_file or with default google credentialsFixes to docstrings and metadataAdd new functionrhg_compute_tools.gcs.rmto remove files/directories on GCS using thegoogle.cloud.storageAPIStore one additional environment variable when passingcred_pathtorhg_compute_tools.kubernetes.get_clusterso that thegoogle.cloud.storageAPI will be authenticated in addition togsutilv0.1.8Deployment fixesv0.1.7Design tools: use RHG & CIL colors & stylesPlotting helpers: generate cmaps with consistent colors & norms, and apply a colorbar to geopandas plots with nonlinear normsAutoscaling fix for kubecluster: switch to dask_kubernetes.KubeCluster to allow use of recent bug fixesv0.1.6Addrhg_compute_tools.gcs.cp_gcsandrhg_compute_tools.gcs.sync_gcsutilitiesv0.1.5need to figure out how to use this rever thingv0.1.4Bug fix again inrhg_compute_tools.kubernetes.get_workerv0.1.3Bug fix inrhg_compute_tools.kubernetes.get_workerv0.1.2Add xarray from delayed methods inrhg_compute_tools.xarrayrhg_compute_tools.gcs.cp_to_gcsnow callsgsutilin a subprocess instead ofgoogle.storageoperations. This dramatically improves performance when transferring large numbers of small filesAdditional cluster creation helpersv0.1.1New google compute helpers (seerhg_compute_tools.gcs.cp_to_gcs,rhg_compute_tools.gcs.get_bucket)New cluster creation helper (seerhg_compute_tools.kubernetes.get_worker)Dask client.map helpers (seerhg_compute_tools.utils submodule)v0.1.0First release on PyPI.