internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: NVIDIA/NVFlare

Issue:
log_exception error
This error happens during log_exception
```
nvflare/private/event.py", line 62, in fire_event
    h.log_exception(
TypeError: log_exception() got an unexpected keyword argument 'local_logging'
FL client execution exception: log_exception() got an unexpected keyword argument 'local_logging'
2022-03-17 18:29:04,424 - ProcessExecutor - INFO - process finished with execution code: 0
```


Discussion:

Resolving patch:
diff --git a/nvflare/private/event.py b/nvflare/private/event.py
--- a/nvflare/private/event.py
+++ b/nvflare/private/event.py
@@ -59,8 +59,6 @@ def fire_event(event: str, handlers: list, ctx: FLContext):
                 ctx.set_prop(key=FLContextKey.EVENT_SCOPE, value=event_scope, private=True, sticky=False)
                 h.handle_event(event, ctx)
             except:
-                h.log_exception(
-                    ctx, 'exception when handling event "{}"'.format(event), fire_event=False, local_logging=True
-                )
+                h.log_exception(ctx, 'exception when handling event "{}"'.format(event), fire_event=False)
 
     ctx.set_prop(key=_KEY_EVENT_DEPTH, value=depth, private=True, sticky=False)
diff --git a/nvflare/widgets/fed_event.py b/nvflare/widgets/fed_event.py
--- a/nvflare/widgets/fed_event.py
+++ b/nvflare/widgets/fed_event.py
@@ -211,7 +211,7 @@ def handle_event(self, event_type: str, fl_ctx: FLContext):
 
     def fire_and_forget_request(self, request: Shareable, fl_ctx: FLContext, targets=None):
         if not self.ready:
-            self.log_warning(fl_ctx, "Engine in not ready, skip the aux event firing.", local_logging=True)
+            self.log_warning(fl_ctx, "Engine in not ready, skip the aux event firing.")
             return
 
         if not isinstance(self.engine, ClientEngineSpec):


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Fix docstring in FLComponent
https://github.com/NVIDIA/NVFlare/blob/main/nvflare/apis/fl_component.py#L32

aggregrators should be aggregators


Discussion:

Resolving patch:
diff --git a/nvflare/apis/fl_component.py b/nvflare/apis/fl_component.py
--- a/nvflare/apis/fl_component.py
+++ b/nvflare/apis/fl_component.py
@@ -30,7 +30,7 @@ def __init__(self):
         """Init FLComponent.
 
         The FLComponent is the base class of all FL Components.
-        (executors, controllers, responders, filters, aggregrators, and widgets are all FLComponents)
+        (executors, controllers, responders, filters, aggregators, and widgets are all FLComponents)
 
         FLComponents have the capability to handle and fire events and contain various methods for logging.
         """


Test patch:
Repository: NVIDIA/NVFlare

Issue:
json files created by provision are not human readable



Discussion:

Resolving patch:
diff --git a/nvflare/lighter/impl/static_file.py b/nvflare/lighter/impl/static_file.py
--- a/nvflare/lighter/impl/static_file.py
+++ b/nvflare/lighter/impl/static_file.py
@@ -145,7 +145,7 @@ def _build_server(self, server, ctx):
             config["overseer_agent"] = overseer_agent
         if self.snapshot_persistor:
             config["snapshot_persistor"] = self.snapshot_persistor
-        self._write(os.path.join(dest_dir, "fed_server.json"), json.dumps(config), "t")
+        self._write(os.path.join(dest_dir, "fed_server.json"), json.dumps(config, sort_keys=True, indent=2), "t")
         replacement_dict = {
             "admin_port": admin_port,
             "fed_learn_port": fed_learn_port,
@@ -213,7 +213,7 @@ def _build_client(self, client, ctx):
             overseer_agent.pop("overseer_exists", None)
             config["overseer_agent"] = overseer_agent
 
-        self._write(os.path.join(dest_dir, "fed_client.json"), json.dumps(config), "t")
+        self._write(os.path.join(dest_dir, "fed_client.json"), json.dumps(config, sort_keys=True, indent=2), "t")
         if self.docker_image:
             self._write(
                 os.path.join(dest_dir, "docker.sh"),
@@ -274,7 +274,7 @@ def _build_admin(self, admin, ctx):
             overseer_agent.pop("overseer_exists", None)
             agent_config["overseer_agent"] = overseer_agent
         config["admin"].update(agent_config)
-        self._write(os.path.join(dest_dir, "fed_admin.json"), json.dumps(config), "t")
+        self._write(os.path.join(dest_dir, "fed_admin.json"), json.dumps(config, sort_keys=True, indent=2), "t")
         if self.docker_image:
             self._write(
                 os.path.join(dest_dir, "docker.sh"),


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Users are not warned when running poc command



Discussion:

Resolving patch:
diff --git a/nvflare/lighter/poc.py b/nvflare/lighter/poc.py
--- a/nvflare/lighter/poc.py
+++ b/nvflare/lighter/poc.py
@@ -62,6 +62,7 @@ def main():
                     os.chmod(os.path.join(root, file), 0o755)
         clone_client(args.num_clients)
         print("Successfully creating poc folder.  Please read poc/Readme.rst for user guide.")
+        print("\n\nWARNING:\n******* Files generated by this poc command are NOT intended for production environments.")
 
 
 if __name__ == "__main__":


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Yaml loader should be replaced by safe_loader or other more secure loader
To load yaml files from unknown source, we should avoid using yaml's loader.  A better way is to use either safe_loader or other mechanism.


Discussion:

Resolving patch:
diff --git a/nvflare/lighter/provision.py b/nvflare/lighter/provision.py
--- a/nvflare/lighter/provision.py
+++ b/nvflare/lighter/provision.py
@@ -21,10 +21,9 @@
 import sys
 import webbrowser
 
-import yaml
-
 from nvflare.fuel.utils.class_utils import instantiate_class
 from nvflare.lighter.spec import Participant, Project, Provisioner
+from nvflare.lighter.utils import load_yaml
 
 
 def main():
@@ -77,7 +76,7 @@ def main():
     project_full_path = os.path.join(current_path, project_file)
     print(f"Project yaml file: {project_full_path}.")
 
-    project_dict = yaml.load(open(project_full_path, "r"), Loader=yaml.Loader)
+    project_dict = load_yaml(project_full_path)
     api_version = project_dict.get("api_version")
     if api_version not in [3]:
         raise ValueError(f"API version expected 3 but found {api_version}")
diff --git a/nvflare/lighter/utils.py b/nvflare/lighter/utils.py
--- a/nvflare/lighter/utils.py
+++ b/nvflare/lighter/utils.py
@@ -46,7 +46,7 @@ def sign_all(content_folder, signing_pri_key):
 
 
 def load_yaml(file_name):
-    return yaml.load(open(file_name, "r"), Loader=yaml.Loader)
+    return yaml.safe_load(open(file_name, "r"))
 
 
 def sh_replace(src, mapping_dict):


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Yaml loader should be replaced by safe_loader or other more secure loader
To load yaml files from unknown source, we should avoid using yaml's loader.  A better way is to use either safe_loader or other mechanism.


Discussion:

Resolving patch:
diff --git a/nvflare/ha/overseer/utils.py b/nvflare/ha/overseer/utils.py
--- a/nvflare/ha/overseer/utils.py
+++ b/nvflare/ha/overseer/utils.py
@@ -16,7 +16,7 @@
 import uuid
 from datetime import datetime, timedelta
 
-import yaml
+from nvflare.lighter.utils import load_yaml
 
 OVERSEER_STORE = os.environ.get("OVERSEER_STORE")
 
@@ -34,7 +34,7 @@
 def load_privilege():
     privilege_file = os.environ.get("AUTHZ_FILE", "privilege.yml")
     try:
-        privilege = yaml.load(open(privilege_file, "tr"), Loader=yaml.Loader)
+        privilege = load_yaml(privilege_file)
     except:
         privilege = dict()
     return privilege
diff --git a/nvflare/lighter/study.py b/nvflare/lighter/study.py
--- a/nvflare/lighter/study.py
+++ b/nvflare/lighter/study.py
@@ -19,7 +19,7 @@
 import os
 from datetime import datetime
 
-import yaml
+from nvflare.lighter.utils import load_yaml
 
 
 def get_input(prompt, item_list, multiple=False):
@@ -75,7 +75,7 @@ def main():
         print(f"{project_full_path} not found.  Running study requires that file.")
         exit(0)
 
-    project = yaml.load(open(project_full_path, "r"), Loader=yaml.Loader)
+    project = load_yaml(project_full_path)
     api_version = project.get("api_version")
     if api_version not in [3]:
         raise ValueError(f"API version expected 3 but found {api_version}")


Test patch:
Repository: NVIDIA/NVFlare

Issue:
[BUG] Tenseal necessary, makes installation impossible on Apple Silicon.

As discussed in #130 tenseal remains unavailable for Apple Silicon. The current NVFlare version (2.2.0) has no optional features and tenseal is necessary, making installation impossible on Apple Silicon.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -85,10 +85,10 @@
         "psutil==5.9.1",
         "PyYAML==6.0",
         "six>=1.15.0",
-        "tenseal==0.3.0",
         "msgpack==1.0.3",
         "docker>=6.0",
     ],
+    extras_require={"HE": ["tenseal==0.3.0"]},
     entry_points={
         "console_scripts": [
             "provision=nvflare.lighter.provision:main",


Test patch:
Repository: NVIDIA/NVFlare

Issue:
StatisticsExecutor init_failure is set instead of dict
https://github.com/NVIDIA/NVFlare/blob/40deb65069907b2a2eb31631f7271a9355c00bb1/nvflare/app_common/executors/statistics/statistics_executor.py#L94: should be `self.init_failure = {"abort_job": te}`
Same in line 97

This causes this error message:
```
Traceback (most recent call last):
  File "/home/hroth/Code2/nvflare/sim-project/NVFlare/nvflare/private/fed/client/client_runner.py", line 236, in _process_task
    reply = executor.execute(task.name, task.data, fl_ctx, self.task_abort_signal)
  File "/home/hroth/Code2/nvflare/sim-project/NVFlare/nvflare/app_common/executors/statistics/statistics_executor.py", line 100, in execute
    init_rc = self._check_init_status(fl_ctx)
  File "/home/hroth/Code2/nvflare/sim-project/NVFlare/nvflare/app_common/executors/statistics/statistics_executor.py", line 165, in _check_init_status
    reason = self.init_failure[fail_key]
TypeError: 'set' object is not subscriptable
```


Discussion:

Resolving patch:
diff --git a/nvflare/app_common/executors/statistics/statistics_executor.py b/nvflare/app_common/executors/statistics/statistics_executor.py
--- a/nvflare/app_common/executors/statistics/statistics_executor.py
+++ b/nvflare/app_common/executors/statistics/statistics_executor.py
@@ -91,10 +91,10 @@ def initialize(self, fl_ctx: FLContext):
                 f"but {type(self.stats_generator).__name__} is provided. {te}",
             )
             self.init_status_ok = False
-            self.init_failure = {"abort_job", te}
+            self.init_failure = {"abort_job": te}
         except Exception as e:
             self.init_status_ok = False
-            self.init_failure = {"fail_client", e}
+            self.init_failure = {"fail_client": e}
 
     def execute(self, task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) -> Shareable:
         init_rc = self._check_init_status(fl_ctx)


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Remove self.server_meta in fed_client_base.py
This is not used: https://github.com/NVIDIA/NVFlare/blob/main/nvflare/private/fed/client/fed_client_base.py#L285


Discussion:
@yhwen can you handle this

Resolving patch:
diff --git a/nvflare/private/fed/client/fed_client_base.py b/nvflare/private/fed/client/fed_client_base.py
--- a/nvflare/private/fed/client/fed_client_base.py
+++ b/nvflare/private/fed/client/fed_client_base.py
@@ -35,6 +35,13 @@
 from .communicator import Communicator
 
 
+def _check_progress(remote_tasks):
+    if remote_tasks[0] is not None:
+        return True, remote_tasks[0].task_name
+    else:
+        return False, None
+
+
 class FederatedClientBase:
     """The client-side base implementation of federated learning.
 
@@ -272,12 +279,8 @@ def pull_task(self, fl_ctx: FLContext):
         try:
             pool = ThreadPool(len(self.servers))
             self.remote_tasks = pool.map(partial(self.fetch_execute_task, fl_ctx=fl_ctx), tuple(self.servers))
-            pull_success, task_name = self.check_progress(self.remote_tasks)
-            # # Update app_ctx's current round info
-            # if self.app_context and self.remote_models[0] is not None:
-            #     self.app_context.global_round = self.remote_models[0].meta.current_round
+            pull_success, task_name = _check_progress(self.remote_tasks)
             # TODO: if some of the servers failed
-            # return self.model_manager.assign_current_model(self.remote_models)
             return pull_success, task_name, self.remote_tasks
         finally:
             if pool:
@@ -360,10 +363,3 @@ def close(self):
             self.overseer_agent.end()
 
         return 0
-
-    def check_progress(self, remote_tasks):
-        if remote_tasks[0] is not None:
-            self.server_meta = remote_tasks[0].meta
-            return True, remote_tasks[0].task_name
-        else:
-            return False, None


Test patch:
Repository: NVIDIA/NVFlare

Issue:
[BUG] Command "nvflare" not found 
Unfortunately I can't run nvflare on Ubuntu because it tells me that the command (`nvflare simulator -h`) is not found. Even if I use the Docker solution. Inside the docker it also shows "command not found". Thus unable to run local experiments.

I tried the MWE from https://nvflare.readthedocs.io/en/2.2.1/getting_started.html

Used OS: Ubuntu 18.04.5 LTS



Discussion:
Hi @hackenjoe Could you please provide which version of NVFlare are you using? Did you install the NVFlare using "pip"? And did you run the command in a virtual environment?
Sure, I use the newest versions. And I installed nvflare using pip running in a new conda environment.

`Name: nvflare
Version: 2.1.2
Summary: Federated Learning Application Runtime Environment
Home-page: https://github.com/NVIDIA/NVFlare
Author: 
Author-email: 
License: 
Location: /opt/conda/lib/python3.8/site-packages
Requires: cryptography, flask, google-api-python-client, grpcio, gunicorn, numpy, psutil, PyYAML, tenseal
Required-by: `

But as mentioned previously, I am using the docker container and inside the container it says that the package is already be installed but yet, the command is not found.

`bash: nvflare: command not found`

EDIT: Want to mention, that I upgraded the OS in the meantime (and still receive same error):

```Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal
```

I have the same issue using the docker container following the Getting Started instructions:
https://nvflare.readthedocs.io/en/2.2.1/getting_started.html
@hackenjoe - as this bug is "closed": do you have a solution to share? Thank you!
@andreaskopp thanks for your interest and question.

I've noticed that our https://nvflare.readthedocs.io/en/2.2.1/getting_started.html needs to be updated.

Please use the following Dockerfile for building the docker image:

```Dockerfile
ARG PYTORCH_IMAGE=nvcr.io/nvidia/pytorch:22.12-py3
FROM ${PYTORCH_IMAGE}

RUN python3 -m pip install -U pip
RUN python3 -m pip install -U setuptools
RUN python3 -m pip install nvflare

WORKDIR /workspace/
RUN git clone https://github.com/NVIDIA/NVFlare.git --branch 2.2 --single-branch NVFlare


```

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -76,6 +76,7 @@ def resolve_xref(self, env, fromdocname, builder, typ, target, node, contnode):
     "sphinx.ext.autodoc",
     "sphinx.ext.viewcode",
     "sphinx.ext.autosectionlabel",
+    "sphinx_copybutton",
 ]
 
 autoclass_content = "both"


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Dashboard UI not generating server and client startup packages.

Hi,

I have created an empty folder and started the dashboard UI according to the link https://nvflare.readthedocs.io/en/main/user_guide/dashboard_api.html.
e.g. nvflare dashboard -p 8080 --start

In the project admin role, I navigated to browser at localhost:8080 and keyed in the server name without HA and made the project public.
After that, I signed up with an org admin role in which the org admin role created 2 client sites.
Back to the admin role, I approved the org admin role, 2 client sites and freezed the project.
However in the Download tab, the admin role is able to download the admin console package but not the server startup package even though it promoted a password to decrypt the zip file.

I am not sure if I have missed any of the above steps.

For your advice.
Thanks.



Discussion:

Resolving patch:
diff --git a/nvflare/dashboard/application/blob.py b/nvflare/dashboard/application/blob.py
--- a/nvflare/dashboard/application/blob.py
+++ b/nvflare/dashboard/application/blob.py
@@ -150,7 +150,7 @@ def gen_server(key, first_server=True):
         if not project.ha_mode:
             _write(
                 os.path.join(dest_dir, get_csp_start_script_name("azure")),
-                utils.sh_replace(get_csp_template("azure", "svr"), {"server_name": entity.name}),
+                utils.sh_replace(get_csp_template("azure", "svr", template), {"server_name": entity.name}),
                 "t",
                 exe=True,
             )
@@ -255,7 +255,7 @@ def gen_client(key, id):
         _write(os.path.join(dest_dir, "rootCA.pem"), project.root_cert, "b", exe=False)
         _write(
             os.path.join(dest_dir, get_csp_start_script_name("azure")),
-            get_csp_template("azure", "cln"),
+            get_csp_template("azure", "cln", template),
             "t",
             exe=True,
         )


Test patch:
Repository: NVIDIA/NVFlare

Issue:
Bug in prostate_2D example
https://github.com/NVIDIA/NVFlare/blob/8f8f029eeecf58a85d9633357ce1ed4f8f39f655/examples/advanced/prostate/prostate_2D/custom/learners/supervised_monai_prostate_learner.py#L171

`self.transform_valid` is not defined if `cache_rate=0`.


Discussion:

Resolving patch:
diff --git a/examples/advanced/prostate/prostate_2D/custom/learners/supervised_monai_prostate_learner.py b/examples/advanced/prostate/prostate_2D/custom/learners/supervised_monai_prostate_learner.py
--- a/examples/advanced/prostate/prostate_2D/custom/learners/supervised_monai_prostate_learner.py
+++ b/examples/advanced/prostate/prostate_2D/custom/learners/supervised_monai_prostate_learner.py
@@ -168,7 +168,7 @@ def train_config(self, fl_ctx: FLContext):
             )
             self.valid_dataset = Dataset(
                 data=valid_list,
-                transform=self.transform_valid,
+                transform=self.transform,
             )
 
         self.train_loader = DataLoader(


Test patch:
Repository: NVIDIA/NVFlare

Issue:
CIFAR-10 Split Learning: Add results with different amounts of data overlap between clients.
              is possible to hide FLARE specific concept in the executor, I felt we should hide as much as we can the FLARE constructs. and expose as little as possible, so that user felt easy to transition from single machine learning to FL learning. In this case particularly, can we not introduce FLContext(), at this point,  As we are just doing data splitting.

_Originally posted by @chesterxgchen in https://github.com/NVIDIA/NVFlare/pull/1168#discussion_r1109010045_

> This way, it can also be used as an FL component similar to the cifar10 FedAvg [example](https://github.com/NVIDIA/NVFlare/blob/3111a23a95d645ac17afb462a04f722997535210/examples/cifar10/cifar10-sim/job_configs/cifar10_fedavg/cifar10_fedavg/config/config_fed_server.json#L15) for simulation purposes.

For example, you can automate the runs by adding the Cifar10VerticalDataSplitter to your job config and run experiments with different overlaps in parallel.

I might add some more experimental results with different overlaps.            


Discussion:

Resolving patch:
diff --git a/examples/advanced/vertical_federated_learning/cifar10-splitnn/set_intersection_file.py b/examples/advanced/vertical_federated_learning/cifar10-splitnn/set_intersection_file.py
--- a/examples/advanced/vertical_federated_learning/cifar10-splitnn/set_intersection_file.py
+++ b/examples/advanced/vertical_federated_learning/cifar10-splitnn/set_intersection_file.py
@@ -38,6 +38,7 @@ def main():
 
     with open(args.config_file, "w") as f:
         json.dump(config, f, indent=4)
+        f.write("\n")
 
     print(f"Modified {args.config_file} to use INTERSECTION_FILE={config['INTERSECTION_FILE']}")
 
diff --git a/examples/advanced/vertical_federated_learning/cifar10-splitnn/src/psi/__init__.py b/examples/advanced/vertical_federated_learning/cifar10-splitnn/src/psi/__init__.py
new file mode 100644
--- /dev/null
+++ b/examples/advanced/vertical_federated_learning/cifar10-splitnn/src/psi/__init__.py
@@ -0,0 +1,15 @@
+# Copyright (c) 2021-2022, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from psi.cifar10_local_psi import Cifar10LocalPSI


Test patch:
Repository: NVIDIA/NVFlare

Issue:
[BUG] AdminAPI not passing SNI when connecting to the nvflare-server
**Describe the bug**

In our setup with NVFlare we noticed that at some case admin shell wouldn't be able to connect to server, even though the client running in same exact environment / network would be able to connect to our server. After some debugging it turned boiled down to these: 
- In our setup we proxy requests to appropriate server based on [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication)
- Proxy wasn't getting any SNI indication when admin client was connecting

**To Reproduce**

I created a [sample repo](https://github.com/xander-aphe-hatschi/nvflare-sni-repro-case) to obtain logs of the connection initiation for both admin and client.

If we take a look at [client logs](https://github.com/xander-aphe-hatschi/nvflare-sni-repro-case/blob/main/client-connecting.log) and [admin logs](https://github.com/xander-aphe-hatschi/nvflare-sni-repro-case/blob/main/admin-connecting.log) first item missing would be `TLS client extension "server name" (id=0) ...` missing from the admin logs

**Expected behavior**

`AdminAPI` class should pass `SNI` when connecting to service


**Desktop (please complete the following information):**
 - OS: Ubuntu 22.04
 - Python Version 3.10.9
 - NVFlare Version 2.3.0

**Additional context**

There's https://github.com/NVIDIA/NVFlare/pull/1850 that seems to fix this. I might need help figuring out how to test it through in test though (manual testing passes).

(Marked as bug just because feels like client/admin should be on-par in terms of being able to connect with the same network setup)



Discussion:

Resolving patch:
diff --git a/nvflare/fuel/hci/client/api.py b/nvflare/fuel/hci/client/api.py
--- a/nvflare/fuel/hci/client/api.py
+++ b/nvflare/fuel/hci/client/api.py
@@ -739,7 +739,7 @@ def _try_command(self, cmd_ctx: CommandContext):
                 ssl_ctx.load_cert_chain(certfile=self.client_cert, keyfile=self.client_key)
 
                 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
-                    with ssl_ctx.wrap_socket(sock) as ssock:
+                    with ssl_ctx.wrap_socket(sock, server_hostname=sp_host) as ssock:
                         ssock.connect((sp_host, sp_port))
                         self._send_to_sock(ssock, cmd_ctx)
             else:


Test patch: