Repository: NVIDIA/NVFlare

Issue:
Plot the results in cifar10 example
Hi, I ran the experiments (run_2 to run_10 in the cifar10 example) following the steps 3.1-3.6 and would like to plot the results and regenerate the figures you show. However, running this command as advised in step 3.3 to view the results does not work:
  cat ./workspaces/poc_workspace/server/run_2/cross_site_val/cross_site_val.json
Could you advise please on how to regenerate the results after completing all experiments?



Discussion:
Can you elaborate on "does not work"?

These files are stored in your file system, can you go in these directories and see if the cross_site_val.json file exist?

if you ran the secure workspace, the results should be in `./workspaces/secure_workspace`.

The training curves can either be visualized using `tensorboard --logdir=./workspaces/.` or the plotting script under figs. See https://github.com/NVIDIA/NVFlare/tree/main/examples/cifar10#4-results
Thank you for your reply,
Let me elaborate further on the problem I have while trying to generate/plot the results after running the experiments for the cifar10 and prostate examples.

I followed the steps as provided in cifar10 and prostate examples for the POC workspace NOT the secure workspace.
So, in the POC workspace with cifar10 example I have the folder ./workspaces/poc_workspace/ which includes the following folders: admin, server, site-1, site-2, ..., site-8. 

Yes, the files "cross_site_val.json" are there for all run_* sub folders under the server folder as shown below for run_2:
./workspaces/poc_workspace/server/run_2/cross_site_val/cross_site_val.json

events.out.tfevents files are there under site-* folders as shown below for site-1 and run_2:
./workspaces/poc_workspace/site-1/run_2/app_site-1/events.out.tfevents.*

Please advise how to generate the plots for the POC workspace runs. I followed the plotting script "plot_tensorboard_events.py" but it is tailored for the secure_workspace.

It would be appreciated to comment on what modifications are required to implement to plot the results. I have the same problem with the prostate example as well where even the plotting script is not there in the figs folder.

One more question is: where can I find all trained models (the local ones at different sites and the global one)? Are there  stored h5 files for the trained models somewhere in the folders?

Thank you!
> Can you elaborate on "does not work"?
> 
> These files are stored in your file system, can you go in these directories and see if the cross_site_val.json file exist?

Thank you! I added a detailed comment if you could check it and advise please.
The plot script is really just used to make the plots for this [readme](https://github.com/NVIDIA/NVFlare/blob/main/examples/cifar10/README.md). It's not a standard tool that is expected to work over all examples.
You should be able to run it on the poc results by setting these [lines](https://github.com/NVIDIA/NVFlare/blob/b5d6226f316e574648bccabf0b043b1d605ffb81/examples/cifar10/figs/plot_tensorboard_events.py#L24) to

```
client_results_root = "./workspaces/poc_workspace/site-1"
server_results_root = "./workspaces/poc_workspace/server"
```
The reason, the script is set up for the secure workspace is that it also plots the results for the HE experiment. HE requires the secure workspace. The plot script should work out of the box if you generate all results using [./run_experiments.sh](https://github.com/NVIDIA/NVFlare/blob/main/examples/cifar10/run_experiments.sh) (after creating the secure workspace).
Also, you need to specify the runs that should be plotted, i.e. remove this [line](https://github.com/NVIDIA/NVFlare/blob/b5d6226f316e574648bccabf0b043b1d605ffb81/examples/cifar10/figs/plot_tensorboard_events.py#L31) if you don't have those results.
Regarding your question about the saved models. You should see *.pt files in both the server and client run folders.
After running all experiments with the command (./run_experiments.sh) in the secure workspace, I am getting the below error when trying to plot the results in part 4.1 Central vs. Fedavg {run_1, run_2, run_9} following the ./figs/plot_tensorboard_events.py script.

Could you help fix this please?

![Capture](https://user-images.githubusercontent.com/76130269/156814435-769db07d-14a4-4232-aacb-c3822b2317ca.PNG)

Resolving patch:
diff --git a/examples/cifar10/figs/plot_tensorboard_events.py b/examples/cifar10/figs/plot_tensorboard_events.py
--- a/examples/cifar10/figs/plot_tensorboard_events.py
+++ b/examples/cifar10/figs/plot_tensorboard_events.py
@@ -76,7 +76,7 @@ def main():
     data = {"Config": [], "Step": [], "Accuracy": []}
 
     if add_cross_site_val:
-        xsite_keys = ["SRV_server", "SRV_server_best"]
+        xsite_keys = ["SRV_FL_global_model.pt", "SRV_best_FL_global_model.pt"]
         xsite_data = {"Config": []}
         for k in xsite_keys:
             xsite_data.update({k: []})
@@ -94,7 +94,7 @@ def main():
 
         if add_cross_site_val:
             xsite_file = glob.glob(
-                os.path.join(server_results_root, exp["run"] + "/**/cross_site_val.json"), recursive=True
+                os.path.join(server_results_root, exp["run"] + "/**/cross_val_results.json"), recursive=True
             )
             assert len(xsite_file) == 1, "No unique x-site file found!"
             with open(xsite_file[0], "r") as f:


Test patch: