Repository: NVIDIA/NVFlare

Issue:
Potential bug in SVT?
Hi all, 
is this intended that in [line 103](https://github.com/NVIDIA/NVFlare/blob/e5e8c15a3455c81cb16268fbaafeef3f26d2b195/nvflare/app_common/filters/svt_privacy.py#L103) of the SVTPrivacy class, only a random choice of the accepted parameters is taken with replacement? This will lead to a lower number of accepted and candidate parameters ([Line 108](https://github.com/NVIDIA/NVFlare/blob/e5e8c15a3455c81cb16268fbaafeef3f26d2b195/nvflare/app_common/filters/svt_privacy.py#L108)) than originally computed. 

So even if I am running SVT it with a fraction of 1.0, not all of my noisy values (3.491.530) are transferred, but around 1.284.489 of them are set to 0. 



Discussion:
Thanks for raising the issue, we will take a look. @YuanTingHsieh  can you look into this. 
Thanks for tagging me, @holgerroth and @ZiyueXu77 can you check if the logic is intended as pointed out by @LSnyd ?
Thanks
Thanks for reporting, what would be the expected behaviour in this case for your experiments?
Hi @wyli,
I would expect that if I am adding SVT with a fraction of 1.0 that all of my noisy values (3.491.530) are transferred. Instead, around 1.284.489 of them are set to 0 due to the random choice with replacement.

Resolving patch:
diff --git a/nvflare/app_common/filters/svt_privacy.py b/nvflare/app_common/filters/svt_privacy.py
--- a/nvflare/app_common/filters/svt_privacy.py
+++ b/nvflare/app_common/filters/svt_privacy.py
@@ -23,7 +23,9 @@
 
 
 class SVTPrivacy(DXOFilter):
-    def __init__(self, fraction=0.1, epsilon=0.1, noise_var=0.1, gamma=1e-5, tau=1e-6, data_kinds: [str] = None):
+    def __init__(
+        self, fraction=0.1, epsilon=0.1, noise_var=0.1, gamma=1e-5, tau=1e-6, data_kinds: [str] = None, replace=True
+    ):
         """Implementation of the standard Sparse Vector Technique (SVT) differential privacy algorithm.
 
         lambda_rho = gamma * 2.0 / epsilon
@@ -35,6 +37,8 @@ def __init__(self, fraction=0.1, epsilon=0.1, noise_var=0.1, gamma=1e-5, tau=1e-
             noise_var (float, optional): additive noise. Defaults to 0.1.
             gamma (float, optional): Defaults to 1e-5.
             tau (float, optional): Defaults to 1e-6.
+            data_kinds (str, optional): Defaults to None.
+            replace (bool): whether to sample with replacement. Defaults to True.
         """
         if not data_kinds:
             data_kinds = [DataKind.WEIGHT_DIFF, DataKind.WEIGHTS]
@@ -47,6 +51,7 @@ def __init__(self, fraction=0.1, epsilon=0.1, noise_var=0.1, gamma=1e-5, tau=1e-
         self.eps_3 = noise_var
         self.gamma = gamma
         self.tau = tau
+        self.replace = replace
 
     def process_dxo(self, dxo: DXO, shareable: Shareable, fl_ctx: FLContext) -> Union[None, DXO]:
         """Compute the differentially private SVT.
@@ -100,7 +105,7 @@ def process_dxo(self, dxo: DXO, shareable: Shareable, fl_ctx: FLContext) -> Unio
             accepted += candidate_idx[above_threshold].tolist()
             candidate_idx = candidate_idx[~above_threshold]
             self.log_info(fl_ctx, "selected {} responses, requested {}".format(len(accepted), n_upload))
-        accepted = np.random.choice(accepted, size=np.int64(n_upload))
+        accepted = np.random.choice(accepted, size=np.int64(n_upload), replace=self.replace)
         # eps_3 return with noise
         noise = np.random.laplace(scale=self.gamma * 2.0 / self.eps_3, size=accepted.shape)
         self.log_info(fl_ctx, "noise max: {}, median {}".format(np.max(np.abs(noise)), np.median(np.abs(noise))))


Test patch: