Package: python-grid5000
python-grid5000is a python package wrapping the Grid’5000 REST API. You can
use it as a library in your python project or you can explore the Grid’5000
resources interactively using the embedded shell.WarningThe code is currently being developed heavily. Jump to the contributing section
if you want to be involved.1 API compatibilityClient versionAPI version1.x3.x (stable)2 ThanksThe core code is borrowed frompython-gitlabwith small adaptations to
conform with the Grid5000 API models (with an ’s’!)3 ContributingTo contribute, you can drop me an email or open an issue for a bug report, or feature request.There are many areas where this can be improved some of them are listed here:The complete coverage of the API isn’t finished (yet) but this should be fairly easy to reach.
Most of the logic go in`grid5000.objects<https://gitlab.inria.fr/msimonin/python-grid5000/blob/master/grid5000/objects.py>`_. And to be honnest I only
implemented the feature that I needed the most.Returnedstatus codearen’t yet well treated.4 Comparison with …RESTfully:
It consumes REST API following theHATEOASprinciples. This allows the client
to fully discover the resources and actions available. Most of the G5K API
follow theses principles but, for instance theStorage APIdon’t. Thus
RESTfully isn’t compatible with all the features offered by the Grid’5000 API.
It’s a ruby library. Python-grid5000 borrows the friendly syntax for resource
browsing, but in python.Execo:
Written in Python. The api module gathers a lot of utils functions leveraging
the Grid’5000 API. Resources aren’t exposed in a syntax friendly manner,
instead functions for some classical operations are exposed (mainly getters).
It has a convenient way of caching the reference API. Python-grid5000 is a
wrapper around the Grid’5000 that seeks 100% coverage. Python-grid5000 is
resource oriented.Raw requests:Thereference for HTTP library in python. Good for prototyping but low-level.
python-grid5000 encapsulates this library.5 Installation and examplesPlease refer tohttps://api.grid5000.fr/doc/4.0/reference/spec.htmlfor
the complete specification.All the examples are exported in the examples subdirectory so you can
easily test and adapt them.The configuration is read from a configuration file located in the home
directory (should be compatible with the restfully one). It can be created
with the following:When accessing the API from outside of Grid’5000 (e.g your local workstation),
you need to specify the following configuration file:echo '
username: MYLOGIN
password: MYPASSWORD
' > ~/.python-grid5000.yamlWhen accessing the API from a Grid’5000 frontend, providing the username
and password is optionnal. Authentication should work out-of-the-box; if
it fails, try updating python-grid5000, or specify the path to the
certificate to use:echo '
verify_ssl: /etc/ssl/certs/ca-certificates.crt
' > ~/.python-grid5000.yamlUsing a virtualenv is recommended (python 3.5+ is required)virtualenv -p python3 venv
source venv/bin/activate
pip install python-grid50005.1 Grid’5000 shellIf you callgrid5000on the command line you should land in a ipython shell.
Before starting, the file$HOME/.python-grid5000.yamlwill be loaded.$) grid5000

Python 3.6.5 (default, Jun 17 2018, 21:32:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: gk.sites.list()
Out[1]:
[<Site uid:grenoble>,
 <Site uid:lille>,
 <Site uid:luxembourg>,
 <Site uid:lyon>,
 <Site uid:nancy>,
 <Site uid:nantes>,
 <Site uid:rennes>,
 <Site uid:sophia>]

In [2]: # gk is your entry point5.2 Reference API5.2.1 Get node informationimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)node_info=gk.sites["nancy"].clusters["grisou"].nodes["grisou-1"]print("grisou-1 has{threads}threads and has{ram}bytes of RAM".format(threads=node_info.architecture["nb_threads"],ram=node_info.main_memory["ram_size"]))5.2.2 Get Versions of resourcesimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)root_versions=gk.root.get().versions.list()print(root_versions)rennes=gk.sites["rennes"]site_versions=rennes.versions.list()print(site_versions)cluster=rennes.clusters["paravance"]cluster_versions=cluster.versions.list()print(cluster_versions)node_versions=cluster.nodes["paravance-1"]print(node_versions)5.2.3 Browse the reference API offlineNote that only GET like requests are accepted on the ref API.importloggingimportjsonfrompathlibimportPathimportosfromgrid5000importGrid5000,Grid5000Offlinelogging.basicConfig(level=logging.DEBUG)# First get a copy of the reference api# This is a one time and out-of-band process,# here we get it by issuing a regular HTTP requestconf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)data=gk.dump_ref_api()Path("ref.yaml").write_text(json.dumps(data))# you can dump the data to a file# and reuse it offline using the dedicated client# here we reuse directly the data we got (no more HTTP requests will be issued)ref=Grid5000Offline(json.loads(Path("ref.yaml").read_text()))print(ref.sites["rennes"].clusters["paravance"].nodes["paravance-1"])5.3 Monitoring API5.3.1 Get Statuses of resourcesimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)rennes=gk.sites["rennes"]site_statuses=rennes.status.list()print(site_statuses)cluster=rennes.clusters["paravance"]cluster_statuses=cluster.status.list()5.4 Job API5.4.1 Job filteringimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)# state=running will be placed in the query paramsrunning_jobs=gk.sites["rennes"].jobs.list(state="running")print(running_jobs)# get a specific job by its uidjob=gk.sites["rennes"].jobs.get("424242")print(job)# or using the bracket notationjob=gk.sites["rennes"].jobs["424242"]print(job)5.4.2 Submit a jobimportloggingimportosimporttimefromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)# This is equivalent to gk.sites.get("rennes")site=gk.sites["rennes"]job=site.jobs.create({"name":"pyg5k","command":"sleep 3600"})whilejob.state!="running":job.refresh()print("Waiting for the job [%s] to be running"%job.uid)time.sleep(10)print(job)print("Assigned nodes :%s"%job.assigned_nodes)5.5 Deployment API5.5.1 Deploy an environmentimportloggingimportosimporttimefromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)# This is equivalent to gk.sites.get("rennes")site=gk.sites["rennes"]job=site.jobs.create({"name":"pyg5k","command":"sleep 3600","types":["deploy"]})whilejob.state!="running":job.refresh()print("Waiting the job [%s] to be running"%job.uid)time.sleep(10)print("Assigned nodes :%s"%job.assigned_nodes)deployment=site.deployments.create({"nodes":job.assigned_nodes,"environment":"debian9-x64-min"})# To get SSH access to your nodes you can pass your public key## from pathlib import Path## key_path = Path.home().joinpath(".ssh", "id_rsa.pub")### deployment = site.deployments.create({"nodes": job.assigned_nodes,#                                       "environment": "debian9-x64-min"#                                       "key": key_path.read_text()})whiledeployment.status!="terminated":deployment.refresh()print("Waiting for the deployment [%s] to be finished"%deployment.uid)time.sleep(10)print(deployment.result)5.6 Storage API5.6.1 Get Storage accessesimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)print(gk.sites["rennes"].storage["home"].access["msimonin"].rules.list())5.6.2 Set storage accesses (e.g for vms)fromnetaddrimportIPNetworkimportloggingimportosimporttimefromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)site=gk.sites["rennes"]job=site.jobs.create({"name":"pyg5k","command":"sleep 3600","resources":"slash_22=1+nodes=1"})whilejob.state!="running":job.refresh()print("Waiting the job [%s] to be running"%job.uid)time.sleep(5)subnet=job.resources_by_type['subnets'][0]ip_network=[str(ip)foripinIPNetwork(subnet)]# create acces for all ips in the subnetaccess=site.storage["home"].access["msimonin"].rules.create({"ipv4":ip_network,"termination":{"job":job.uid,"site":site.uid}})# listing the accessesprint(gk.sites["rennes"].storage["home"].access["msimonin"].rules.list())5.7 Vlan API5.7.1 Get vlan(s)importloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)site=gk.sites["rennes"]# Get all vlansvlans=site.vlans.list()print(vlans)# Get on specificvlan=site.vlans.get("4")print(vlan)vlan=site.vlans["4"]print(vlan)# Get vlan of some nodesprint(site.vlansnodes.submit(["paravance-1.rennes.grid5000.fr","paravance-2.rennes.grid5000.fr"]))# Get nodes in vlanprint(site.vlans["4"].nodes.list())5.7.2 Set nodes in vlanPutting primary interface in a vlanimportloggingimportosimporttimefromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)site=gk.sites["rennes"]job=site.jobs.create({"name":"pyg5k","command":"sleep 3600","resources":"{type='kavlan'}/vlan=1+nodes=1","types":["deploy"]})whilejob.state!="running":job.refresh()print("Waiting the job [%s] to be runnning"%job.uid)time.sleep(5)deployment=site.deployments.create({"nodes":job.assigned_nodes,"environment":"debian9-x64-min","vlan":job.resources_by_type["vlans"][0]})whiledeployment.status!="terminated":deployment.refresh()print("Waiting for the deployment [%s] to be finished"%deployment.uid)time.sleep(10)print(deployment.result)Putting the secondary interface in a vlanimportloggingimportosimporttimefromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)def_to_network_address(host,interface):"""Translate a host to a network address
    e.g:
    paranoia-20.rennes.grid5000.fr -> paranoia-20-eth2.rennes.grid5000.fr
    """splitted=host.split('.')splitted[0]=splitted[0]+"-"+interfacereturn".".join(splitted)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)site=gk.sites["rennes"]job=site.jobs.create({"name":"pyg5k","command":"sleep 3600","resources":"{type='kavlan'}/vlan=1+{cluster='paranoia'}nodes=1","types":["deploy"]})whilejob.state!="running":job.refresh()print("Waiting the job [%s] to be runnning"%job.uid)time.sleep(5)vlanid=job.resources_by_type["vlans"][0]# we hard code the interface but this can be discovered in the node info# TODO: write the code here to discovernodes=[_to_network_address(n,"eth2")forninjob.assigned_nodes]print(nodes)# set in vlansite.vlans[vlanid].nodes.submit(nodes)5.7.3 Finding vlans of a user, users of a vlanThe Vlan API allows to check which vlans are being used by a user using the
vlanusers manager. Additionally for each individual vlan it is possible to
check whether a user is authorized or not.importosfromgrid5000importGrid5000conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)site=gk.sites["rennes"]#Get list of users using vlansusers=site.vlansusers.list()#Get list of vlans a user is using.user=site.vlansusers['msimonin']print(user.vlans)#Get list of users using a specific vlanusers=site.vlans['4'].users.list()#Check if a user has access to a specific vlan.user=site.vlans['4'].users['msimonin']print(user.status)5.7.4 Vlan StichingThe stitching API allows users to connect a Grid’5000 global vlan to external vlans
connected to other testbeds for experiments involving wide area layer2 networks.conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)# List current stitchingsgk.stitcher.list()# Sitching global vlan 16 to external vlan 1290gk.stitcher.create({"id":"16","sdx_vlan_id":"1290"})# Get a a stitchings information by Grid'5000 global vlan id.stitching=gk.stitcher.get('16')# Orstitching=gk.stitcher['16']# End stitchingstitching=gk.stitcher.get('16')stitching.delete()# Orgk.stitcher.delete('16')5.8 Metrics API5.8.1 Get some timeseries (and plot them)For this example you’ll needmatplotlib,seabornandpandas.importloggingimportosfromgrid5000importGrid5000importpandasaspdimportmatplotlib.pyplotaspltimportseabornassnsimporttimelogging.basicConfig(level=logging.INFO)conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)metrics=gk.sites["lyon"].clusters["nova"].metricsprint("--- available metrics")print(metrics)print("----- a timeserie")now=time.time()# NOTE that you can pass a job_id herekwargs={"nodes":"nova-1,nova-2,nova-3","metrics":"wattmetre_power_watt","start_time":int(now-600),}metrics=gk.sites["lyon"].metrics.list(**kwargs)# let's visualize thisdf=pd.DataFrame()formetricinmetrics:timestamp=metric.timestampvalue=metric.valuedevice_id=metric.device_iddf=pd.concat([df,pd.DataFrame({"timestamp":[timestamp],"value":[value],"device_id":[device_id]})])sns.relplot(data=df,x="timestamp",y="value",hue="device_id",kind="line")plt.show()5.9 More snippets5.9.1 Site of a clusterimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)clusters=["dahu","parasilo","chetemi"]conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)sites=gk.sites.list()matches=[]forsiteinsites:candidates=site.clusters.list()matching=[c.uidforcincandidatesifc.uidinclusters]iflen(matching)==1:matches.append((site,matching[0]))clusters.remove(matching[0])print("We found the following matches%s"%matches)5.9.2 Get all job with a given name on all the sitesimportloggingimportosfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)NAME="pyg5k"conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")gk=Grid5000.from_yaml(conf_file)sites=gk.sites.list()site=gk.sites["rennes"]sites=[gk.sites["rennes"],gk.sites["nancy"],gk.sites["grenoble"]]# creates some jobsjobs=[]forsiteinsites:job=site.jobs.create({"name":"pyg5k","command":"sleep 3600"})jobs.append(job)_jobs=[]forsiteinsites:_jobs.append((site.uid,site.jobs.list(name=NAME,state="waiting,launching,running")))print("We found%s"%_jobs)# deleting the jobsforjobinjobs:job.delete()5.9.3 Caching API responsesThe Grid’5000 reference API is static. In this situation to speed up the
requests, one could leverage heavily on caching. Currentlypython-grid5000doesn’t do caching out-of the box but defers that to the
consuming application. There are many solutions to implement a cache.
Amongst them LRU cache
(https://docs.python.org/3/library/functools.html#functools.lru_cache)
provides an in-memory caching facilities but doesn’t give you control on the
cache. The ring library (https://ring-cache.readthedocs.io/en/stable/) is
great as it implements different backends for your cache (esp.
cross-processes cache) and give you control on the cached object. Enough talking:importloggingimportthreadingimportosimportdiskcachefromgrid5000importGrid5000importring_api_lock=threading.Lock()# Keep track of the api client_api_client=Nonestorage=diskcache.Cache('cachedir')defget_api_client():"""Gets the reference to the API cient (singleton)."""with_api_lock:global_api_clientifnot_api_client:conf_file=os.path.join(os.environ.get("HOME"),".python-grid5000.yaml")_api_client=Grid5000.from_yaml(conf_file)return_api_client@ring.disk(storage)defget_sites_obj():"""Get all the sites."""gk=get_api_client()returngk.sites.list()@ring.disk(storage)defget_all_clusters_obj():"""Get all the clusters."""sites=get_sites_obj()clusters=[]forsiteinsites:# should we cache the list aswell ?clusters.extend(site.clusters.list())returnclustersif__name__=="__main__":logging.basicConfig(level=logging.DEBUG)clusters=get_all_clusters_obj()print(clusters)print("Known key in the cache")print(get_all_clusters_obj.get())print("Calling again the function is now faster")clusters=get_all_clusters_obj()print(clusters)5.9.4 Using Grid’5000 client certificatespython-grid5000can also be used as a trusted client with Grid’5000
internal certificate. In this mode users can pass theg5k_userargument
to most calls to specify which user the API call should be made as. In
cases whereg5k_useris not specified API calls will be made as theanonymoususer whose access is limited to the Grid’5000 reference API.
In this modepython-grid5000does not store any login information, sog5k_usermust be provided explicitly provided on every call that requires
one.importloggingfromgrid5000importGrid5000logging.basicConfig(level=logging.DEBUG)gk=Grid5000(uri="https://api-ext.grid5000.fr/stable/",sslcert="/path/to/ssl/certfile.cert",sslkey="/path/to/ssl/keyfile.key")gk.sites.list()job=site.jobs.create({"name":"pyg5k","command":"sleep 3600"},g5k_user="auser1")# Since the 'anonymous' user can not inspect jobs the following call will raise exception# python-grid5000.exceptions.Grid5000AuthenticationError: 401 Unauthorizedjob.refresh()# Both following call work since any user can request info on any jobs.job.refresh(g5k_user='auser1')job.refresh(g5k_user='auser2')# Some operations can only be performed by the jobs creator.# The following call will raise exception# pyg5k.exceptions.Grid5000DeleteError: 403 Unauthorizedjob.delete(g5k_user='auser2')# This call works as expectedjob.delete(g5k_user='auser1')