Package: gcpm
Google Cloud Platform Condor Pool Manager (GCPM)(Coverage report,Bandit report)HTCondor pool manager for Google Cloud Platform.InstallationPackage installationGCPM can be installed bypip:$ pip install gcpmService file installationTo install as service, do:$ gcpm install:warning: Service installation is valid only for the system managed bySystemd.Iflogrotateis installed, logrotation definition for/var/log/gcpm.logis also installed.Configuration fileConfiguration file pathThe default configuration file is~/.config/gcpm/gcpm.yml.For service, the configuration file is/etc/gcpm.yml.To change the configuration file, use--configoption:$ gcpm run --config /path/to/my/gcpm.ymlConfiguration file contentA configuration file is YAML format.NameDescriptionDefault ValueMandatoryconfig_dirDirectory for some gcpm related files.~/.config/gcpm/(user)/var/cache/gcpm(service)Nooatuh_filePath to OAuth information file for GCE/GCS usage.<config_dir>/oauthNoservice_account_fileService account JSON file for GCE/GCS usage.If not specified, OAuth connection is tried.-NoprojectGoogle Cloud Platform Project Name.-YeszoneZone for Google Compute Engine.-YesmachinesArray of machine settings.Each setting is array of [core, mem, disk, idle, image] (see below).[]Yesmachines:coreNumber of core of the machine type.-Yesmachines:memMemory (MB) of the machine type.-Yesmachines:swapSwap memory (MB) of the machine type.Same as memNomachines:diskDisk size (GB) of the machine type.-Yesmachines:maxLimit of the number of instances for the machine type.-Yesmachines:idleNumber of idle machines for the machine type.-Yesmachines:imageImage of the machine type.-Yesmachines:<others>Other any options can be defined for creating instance.-Nomax_coresLimit of the total number of cores of all instances.If it is set 0, no limit is applied.0Nostatic_wnsArray of instance names of static worker nodes, which are added as condor worker nodes.[]Norequired_machinesArray of machines which should be running other than worker nodes.[]Norequired_machines:nameNumber of core of the machine type.-Yesrequired_machines:memMemory (MB) of the machine type.-Yesrequired_machines:swapSwap memory (MB) of the machine type.Same as memNorequired_machines:diskDisk size (GB) of the machine type.-Yesrequired_machines:imageImage of the machine type.-Yesrequired_machines:<others>Other any options can be defined for creating instance.-Noprimary_accountsUser accounts which jobs must run normal worker nodes. See below about primary accounts.[]NoprefixPrefix of machine names.gcp-wnNopreemptible1 for preemptible machines, 0 for not.0Nooff_timerSecond to send condor_off after starting.0Nostartup_cmdAdditional commands at WN startup.""Noshutdown_cmdAdditional commands at WN shutdown.""Nonetwork_tagArray of GCP network tag.[]Noreuse1 to reused terminated instance. Otherwise delete and re-created instances.0NointervalSecond of interval for each loop.10Noclean_timeTime to clean up residual instances in starting/deleting status.600Nohead_infoIfheadis empty, head node information is automatically taken for each option:hostname: Hostnameip: IP addressgcp: HostnamegcpNoheadHead node Hostname/IP address.""NoportHTCondor port.9618NodomainDomain of the head node.Set empty to take it from hostnaem.""NoadminHTCondor admin email address.""YesownerHTCondor owner name.""Yeswait_cmd1 to wait GCE commands result (create/start/stop/delete...).0NobucketBucket name for pool_password file.""YesstorageClassStorage class name of the bucket."REGIONAL"NolocationStorage location for the bucket.If empty, it is decided from thezone.""Nolog_fileLog file path. Empty to put it in stdout.""Nolog_levelLog level. (debug,info,warning,error,critical)infoNoNote:Primary accountsIf primary accounts are set, jobs ofnon-primaryaccounts can run on test worker nodes.If there are already max number of 1 core worker nodes
and idle jobs of non-primary accounts are there,
test worker node named<prefix>-test-1core-XXXXwill be launched
and only non-primary account jobs can run on it.This able to run such a test job w/o waiting for finishing any normal jobs.Such test worker nodes can be launched until total cores are smaller thanmax_core.To use this function effectively, set total ofmaxof each core to less thanmax_core.e.g.)---
machines:
  core: 1
  max: 10
machines:
  core: 8
  max:  2
max_core: 20
primary_accounts:
  - condor_primaryIn this case, normal jobs can launch 10 1-core machines and 2 8-core machines,
then 16 cores are used.Even if there are a log of idlecondor_primary's jobs,
1 core test jobs by other accounts can run: 4 jobs at most.Puppet setuprcmdnk/puppet-gcpmA puppet module for GCPM.rcmdnk/gcpm-puppetA puppet example to create head (manager) node and worker node with puppet.rcmdnk/frontiersquid-puppetA puppet example to create frontier squid proxy server in GCP.