Project: GraphScope (App Definition and Development / Database)
File: tutorial_dev_algo_cpp_flash.md

We have implemented over 70 graph algorithms with FLASH for over 40 different common problems. These algorithms cover various areas such as clustering, centrality, traversal, matching, mining, etc. We continuously add more algorithms to the FLASH library and encourage users to develop their algorithms using the FLASH model if the built-in algorithms do not meet their requirements. is a programming model specifically designed for writing distributed graph processing algorithms. It follows the Bulk Synchronous Parallel (BSP) computing paradigm and provides primary functions like VSize, VertexMap, and EdgeMap, which are defined on the VertexSubset structure, and a series of auxiliary functions like set operations. The FLASH model provides strong expressiveness and makes it possible to program more advanced algorithms than existing vertex-centric algorithms. To implement your algorithms using the FLASH model, you need to define a FLASH application first, by fulfilling an APP class in C++. If the result of the algorithm is a value on each vertex, the class looks like: ```cpp template <typename FRAG_T> class YourApp : public FlashAppBase<FRAGT, VTYPE> { public: INSTALLFLASHWORKER(YourApp<FRAGT>, VTYPE, FRAG_T) using contextt = FlashVertexDataContext<FRAGT, VTYPE, VRES_TYPE>; bool syncall = false; // true or false VRESTYPE Res(V_TYPE v) { // return the result on the vertex } void Run(const fragmentt& graph, const std::sharedptr<fw_t> fw, ...) { // conduct processing using the FLASH APIs } }; ``` Or, if the result of the algorithm is a single global value, the class looks like: ```cpp template <typename FRAG_T> class YourApp : public FlashAppBase<FRAGT, VTYPE> { public: INSTALLFLASHWORKER(YourApp<FRAGT>, VTYPE, FRAG_T) using contextt = FlashGlobalDataContext<FRAGT, VTYPE, GRES_TYPE>; bool syncall = false; // true or false GRESTYPE Res() { // return the global result } void Run(const fragmentt& graph, const std::sharedptr<fw_t> fw, ...) { // conduct processing using the FLASH APIs } }; ``` After that, a user can implement a distributed graph processing algorithm through utilizing various FLASH APIs in the Run Function of the application. Let's take BFS as an example, an user-defined application may be like this in file `GraphScope/analytical_engine/apps/flash/traversal/bfs.h`: ```cpp template <typename FRAG_T> class BFSFlash : public FlashAppBase<FRAGT, BFSTYPE> { public: INSTALLFLASHWORKER(BFSFlash<FRAGT>, BFSTYPE, FRAG_T) using contextt = FlashVertexDataContext<FRAGT, BFS_TYPE, int>; bool syncall = false; int Res(value_t v) { return &(v->dis); } void Run(const fragmentt& graph, const std::sharedptr<fw_t> fw, oidt osource) { vidt source = Oid2FlashId(osource); DefineMapV(init) { v.dis = (id == source) ? 0 : -1; }; vset_t a = VertexMap(All, CTrueV, init); DefineFV(filter) { return id == source; }; a = VertexMap(a, filter); DefineMapE(update) { d.dis = s.dis + 1; }; DefineFV(cond) { return

== -1; }; for (int len = VSize(a), i = 1; len > 0; len = VSize(a), ++i) { a = EdgeMap(a, ED, CTrueE, update, cond); } } }; ``` As shown in this code, to implement a BFS algorithm, users only need to design and implement processing operations on the VertexSubset, rather than considering communication, message passing and graph partitioning in a distributed setting. This reduces the burden of programming various kinds of algorithms. To run your FLASH algorithms from the Python client of GraphScope, one needs to provide a corresponding Python function defined for this algorithm. For example, to integrate the above BFS algorithm, one can define a function as follows in the file `GraphScope/python/graphscope/analytical/app/flash/traversal.py`. ```python @projecttosimple @notcompatiblefor("arrowproperty", "dynamicproperty") def bfs(graph, source=1): """Evaluate BFS on a graph with flash computation mode. Args: graph (:class:`graphscope.Graph`): A simple graph. source (optional): The source Node. Defaults to 1. Returns: :class:`graphscope.framework.context.Context`: A context with each vertex assigned with the distance from source. """ return AppAssets(algo="flashbfs", context="vertexdata")(graph, source) ``` In addition, the configuration information needs to be added in `GraphScope/coordinator/gscoordinator/builtin/app/.gs_conf.yaml`. Then the algorithm could be invoked from the Python client. ```yaml algo: flash_bfs type: cpp_flash class_name: gs::BFSFlash src: apps/flash/traversal/bfs.h compatible_graph: grape::ImmutableEdgecutFragment gs::ArrowProjectedFragment ``` To run your own FLASH algorithms, you may trigger it from the client of GraphScope once you have installed and deployed GAE. At first, a property graph will be loaded into GraphScope. ```python import graphscope graphscope.setoption(showlog=True) # enable logging ``` ```python from graphscope.dataset import loadp2pnetwork graph = loadp2pnetwork(directed=True) ``` The FLASH algorithms are defined on a simple graph, which has only one kind of vertices and edges, edges and vertices have at most one property as their attribute. Therefore, one needs to project a property graph to convert it to a simple graph, by selecting one kind of label for vertices/edges, and each with at most one of their properties. ```python simple_graph = graph.project(vertices={"host": []}, edges={"connect": ["dist"]}) ``` Then, we can run the BFS algorithm implemented in the FLASH model, by taking the projected graph in the previous step as the input parameter, and setting the source node id to be "1". The algorithm will codegen a compatible version for the loaded graph, and compile to an executable binary. It may take a bit longer to build the library. However, this step only takes once for the same algorithm on a typed graph. ```python context = graphscope.flash.bfs(simple_graph, source=1) ``` After the computation, the results are distributed on the vineyard instances on the cluster. The returned object is a Context, which has several methods to retrieve or persist the results. In this case, the results represent the distance from the source node. We use the following code to fetch the results and display them with their vertex id. ```python context.to_dataframe( selector={"id": "v.id", "dist": "r"}, vertex_range={"begin": 1, "end": 10} ).sort_values(by="id") ```