Project: GraphScope (App Definition and Development / Database)
File: tutorial_node_classification_k8s.md

GraphScope is designed for processing large graphs, which are usually hard to fit in the memory of a single machine. With vineyard as the distributed in-memory data manager, GraphScope supports run on a cluster managed by Kubernetes(k8s). In this tutorial, we revisit the example we present in the first tutorial, showing how GraphScope process the node classification task on a Kubernetes cluster. Please note, since this tutorial is designed to run on a k8s cluster, you need to configure your k8s environment before running the example. ```python import graphscope from graphscope.dataset import loadogbnmag graphscope.setoption(showlog=True) sess = graphscope.session(withdataset=True, k8sservicetype='LoadBalancer', k8simagepullpolicy='Always') ``` Behind the scenes, the session tries to launch a coordinator, which is the entry for the back-end engines. The coordinator manages a cluster of k8s pods (2 pods by default), and learning engines ran on them. For each pod in the cluster, there is a vineyard instance at service for distributed data in memory. The log GraphScope coordinator service connected means the session launches successfully, and the current Python client has connected to the session. You can also check a session's status by this. ```python sess ``` Run this cell, you may find a "status" field with value "active". Together with the status, it also prints other metainfo of this session, i.e., such as the number of workers (pods), the coordinator endpoint for connection, and so on. ```python graph = loadogbnmag(sess, "/dataset/ogbnmagsmall/") print(graph) ``` ```python i_features = [] for i in range(128): ifeatures.append("feat" + str(i)) lg = sess.graphlearn( graph, nodes=[("paper", i_features)], edges=[("paper", "cites", "paper")], gen_labels=[ ("train", "paper", 100, (0, 75)), ("val", "paper", 100, (75, 85)), ("test", "paper", 100, (85, 100)), ], ) try: import tensorflow.compat.v1 as tf tf.disablev2behavior() except ImportError: import tensorflow as tf import argparse import graphscope.learning.graphlearn.python.nn.tf as tfg from graphscope.learning.examples import EgoGraphSAGE from graphscope.learning.examples import EgoSAGEUnsupervisedDataLoader from graphscope.learning.examples.tf.trainer import LocalTrainer def parse_args(): argparser = argparse.ArgumentParser("Train EgoSAGE Unsupervised.") argparser.addargument('--batchsize', type=int, default=128) argparser.addargument('--featuresnum', type=int, default=128) argparser.addargument('--hiddendim', type=int, default=128) argparser.addargument('--outputdim', type=int, default=128) argparser.addargument('--nbrsnum', type=list, default=[5, 5]) argparser.addargument('--negnum', type=int, default=5) argparser.addargument('--learningrate', type=float, default=0.0001) argparser.add_argument('--epochs', type=int, default=1) argparser.addargument('--aggtype', type=str, default="mean") argparser.addargument('--dropout', type=float, default=0.0) argparser.add_argument('--sampler', type=str, default='random') argparser.addargument('--negsampler', type=str, default='in_degree') argparser.add_argument('--temperature', type=float, default=0.07) return argparser.parse_args() args = parse_args() dims = [args.featuresnum] + [args.hiddendim] * (len(args.nbrsnum) - 1) + [args.outputdim] model = EgoGraphSAGE(dims, aggtype=args.aggtype, dropout=args.drop_out) train_data = EgoSAGEUnsupervisedDataLoader(lg, None, sampler=args.sampler, negsampler=args.negsampler, batchsize=args.batchsize, nodetype='paper', edgetype='cites', nbrsnum=args.nbrsnum) srcemb = model.forward(traindata.src_ego) dstemb = model.forward(traindata.dst_ego) negdstemb = model.forward(traindata.negdst_ego) loss = tfg.unsupervisedsoftmaxcrossentropyloss( srcemb, dstemb, negdstemb, temperature=args.temperature) optimizer = tf.train.AdamOptimizer(learningrate=args.learningrate) trainer = LocalTrainer() trainer.train(train_data.iterator, loss, optimizer, epochs=args.epochs) ``` Finally, a session manages the resources in the cluster, thus it is important to release these resources when they are no longer required. To de-allocate the resources, use the method close on the session when all the graph tasks are finished. ```python sess.close() ```