Project: GraphScope (App Definition and Development / Database)
File: persistent_storage_of_graphs_on_k8s.md

If you want to persistently store specific graphs that have been calculated over a long period of time on the Kubernetes cluster and restore them later, this document provides step-by-step instructions on how to do this with the Kubernetes PersistentVolumes. You have a Kubernetes cluster on hand. If you don't have a Kubernetes cluster, please refer to for details. You have the `graphscope` Python library installed. If you don't have installed it, please refer to for details. ```bash $ kubectl create namespace graphscope-system ``` Then create the pv as follows, the pv will be mounted to `/var/vineyard/dump` in the Kubernetes node. You can change the path to any other path you want. ```bash $ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolume metadata: name: graphscope-pv labels: app.kubernetes.io/name: test-pv spec: capacity: storage: 1Gi accessModes: ReadWriteOnce hostPath: path: /var/vineyard/dump storageClassName: manual EOF ``` Create pvc as follows. Most importantly, the pvc can't be deleted, otherwise the data will be lost. ```bash $ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: graphscope-pvc namespace: graphscope-system spec: selector: matchLabels: app.kubernetes.io/name: test-pv resources: requests: storage: 1Gi accessModes: ReadWriteOnce storageClassName: manual EOF ``` After the above preparations are completed, you can deploy the graphscope cluster as follows: ```python import graphscope import os import vineyard from graphscope.dataset import loadmoderngraph k8s_volumes = { "data": { "type": "hostPath", "field": {"path": os.environ["GSTESTDIR"], "type": "Directory"}, "mounts": {"mountPath": "/testingdata"}, } } sess = graphscope.session( num_workers=1, k8simageregistry="docker.io", k8simagetag="ccc", k8s_namespace="graphscope-system", k8svineyarddeployment="vineyardd-sample", k8svolumes=k8svolumes, ) graph = loadmoderngraph(sess, "/testingdata/modern_graph") interactive = sess.gremlin(graph) sub_graph = interactive.subgraph( 'g.V().hasLabel("person").outE("knows")' ) simpleg = subgraph.project(vertices={"person": []}, edges={"knows": []}) prresult = graphscope.pagerank(simpleg, delta=0.8) tcresult = graphscope.triangles(simpleg) subgraph.addcolumn(pr_result, {"Ranking": "r"}) subgraph.addcolumn(tc_result, {"TC": "r"}) print(simpleg.vineyardid) print(subgraph.vineyardid) sess.storegraphsto_pvc( graphIDs=[vineyard.ObjectID(simpleg.vineyardid), vineyard.ObjectID(subgraph.vineyardid)], path="/var/vineyard/dump", pvc_name="graphscope-pvc", ) print(simple_g.schema) print(sub_graph.schema) sess.close() ``` Remember the vineyard ids printed above and the pvc name and then you can restore the graphs from the pvc as follows. ```python import graphscope import os import vineyard sess = graphscope.session( num_workers=1, k8simageregistry="docker.io", k8simagetag="ccc", k8s_namespace="graphscope-system", k8svineyarddeployment="vineyardd-sample", ) sess.restoregraphsfrom_pvc( path="/var/vineyard/dump", pvc_name="graphscope-pvc" ) simple_g = sess.g(vineyard.ObjectID(997255889378630)) sub_graph = sess.g(vineyard.ObjectID(997163552113975)) print(simple_g.schema) print(sub_graph.schema) ``` If you don't need the graphs anymore, you can delete the pvc and pv as follows. ```bash $ kubectl delete pvc graphscope-pvc -n graphscope-system $ kubectl delete pv graphscope-pv ```