Repository: dmlc/dgl
Pull request: [Dist] Create <graph_name>_stats.txt file if it does not exist before ParMETIS execution
## Description
<!-- Brief description. Refer to the related issues if existed.
It'll be great if relevant reviewers can be assigned as well.-->
Here, we check if the <graph_name>_stats.txt does not exist then creates one in the current directory. 
<graph_name>_stats.txt file has one single line with 3 columns (num_nodes, num_edges and num_constraints). 
num_constraints is the sum of number of node types in the graph (represented as 1-hot vector) and one each for train/test/val masks. 

stats file will be created in the current working directory. 

## Checklist
Please feel free to remove inapplicable items for your PR.
- [X] The PR title starts with [$CATEGORY] (such as [NN], [Model], [Doc], [Feature]])
- [X] I've leverage the [tools](https://docs.google.com/document/d/1iHyj7zlmygKSk5gBPsqIqL5ASPzJSPREaNT_QdsiYA4/edit) to beautify the python and c++ code.
- [X] The PR is complete and small, read the [Google eng practice (CL equals to PR)](https://google.github.io/eng-practices/review/developer/small-cls.html) to understand more about small PR. In DGL, we consider PRs with less than 200 lines of core code change are small (example, test and documentation could be exempted).
- [X] All changes have test coverage
- [X] Code is well-documented
- [X] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
- [ ] Related issue is referred in this PR
- [ ] If the PR is for a new model/paper, I've updated the example index [here](../examples/README.md).

## Changes
<!-- You could use following template
- [ ] Feature1, tests, (and when applicable, API doc)
- [ ] Feature2, tests, (and when applicable, API doc)
-->


Problem statement:
Check if <graph_name>_stats.txt exists and if not create one.
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

We cannot expect use to provide "<graph_name>_stats.txt" with the dataset. 
If this file does not exist then the pipeline will create one (in the current working directory). 
This file is needed for ParMETIS to function properly. 

## To Reproduce

Steps to reproduce the behavior:

1. If <graph_name>_stats.txt does not exist ParMETIS will fail to run.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

 - DGL Version (e.g., 1.0):
 - Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):
 - OS (e.g., Linux):
 - How you installed DGL (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version (if applicable):
 - GPU models and configuration (e.g. V100):
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Hints from issue discussion:

Commit message: check if stats file exists, if not create one before parmetis run

Patch to review:
diff --git a/tools/distpartitioning/parmetis_preprocess.py b/tools/distpartitioning/parmetis_preprocess.py
index 4f250d20b7bc..8ea3d4adab60 100644
--- a/tools/distpartitioning/parmetis_preprocess.py
+++ b/tools/distpartitioning/parmetis_preprocess.py
@@ -287,6 +287,36 @@ def gen_parmetis_input_args(params, schema_map):
         schema_map[constants.STR_NUM_NODES_PER_CHUNK],
     )
 
+    # Check if <graph-name>_stats.txt exists, if not create one using metadata.
+    # Here stats file will be created in the current directory. 
+    # No. of constraints, third column in the stats file is computed as follows:
+    #   num_constraints = no. of node types + train_mask + test_mask + val_mask
+    #   Here, (train/test/val) masks will be set to 1 if these masks exist for
+    #   all the node types in the graph, otherwise these flags will be set to 0
+    assert graph_name in schema_map, "Graph name is not present in the json file"
+    graph_name = schema_map[constants.GRAPH_NAME]
+    if not os.path.isfile(f'{graph_name}_stats.txt'):
+        num_nodes = np.sum(np.concatenate(schema_map[constants.STR_NUM_NODES_PER_CHUNK]))
+        num_edges = np.sum(np.concatenate(schema_map[constants.STR_NUM_EDGES_PER_CHUNK]))
+        num_ntypes = len(schema_map[constants.STR_NODE_TYPE])
+
+        train_mask = test_mask = val_mask = 0
+        node_feats = schema_map[constants.STR_NODE_DATA]
+        for ntype, ntype_data in node_feats.items():
+            if "train_mask" in ntype_data:
+                train_mask += 1
+            if "test_mask" in ntype_data:
+                test_mask += 1
+            if "val_mask" in ntype_data:
+                val_mask += 1
+        train_mask = train_mask // num_ntypes
+        test_mask = test_mask // num_ntypes
+        val_mask = val_mask // num_ntypes
+        num_constraints = num_nyptes + train_mask + test_mask + val_mask
+
+        with open(f'{graph_name}_stats.txt', 'w') as sf:
+            sf.write(f'{num_nodes} {num_edges} {num_constraints}')
+
     node_files = []
     outdir = Path(params.output_dir)
     os.makedirs(outdir, exist_ok=True)


Code review comments:
diff_hunk: @@ -287,6 +287,36 @@ def gen_parmetis_input_args(params, schema_map):
         schema_map[constants.STR_NUM_NODES_PER_CHUNK],
     )
 
+    # Check if <graph-name>_stats.txt exists, if not create one using metadata.
+    # Here stats file will be created in the current directory. 
+    # No. of constraints, third column in the stats file is computed as follows:
+    #   num_constraints = no. of node types + train_mask + test_mask + val_mask
+    #   Here, (train/test/val) masks will be set to 1 if these masks exist for
+    #   all the node types in the graph, otherwise these flags will be set to 0
+    assert graph_name in schema_map, "Graph name is not present in the json file"
path: tools/distpartitioning/parmetis_preprocess.py
text: @user1:
where is `graph_name` defined? should be `constants.GRAPH_NAME`?

@author:
graph_name is present in the json file of the input graph dataset. It is one of the keys in the dictionary object created by reading the json file. Per our new dataset format, graph_name should be present in the json file for the input dataset. 

This is a typo and was interchanged the next line. You are right, it should be constants.GRAPH_NAME. Changed it accordingly.