Repository: alibaba/FederatedScope

Issue:
Inappropriate output order
> 2022-12-06 21:22:48,250 (server:329) INFO: Server: Starting evaluation at the end of round 1.
2022-12-06 21:22:48,253 (server:335) INFO: ----------- Starting a new training round (Round #2) -------------
2022-12-06 21:22:49,032 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:49,386 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:50,482 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:50,753 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:51,852 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:52,207 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:53,362 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:53,716 (context:224) WARNING: cur_mode `test` mismatch mode `val`, will use `val` to calculate `ctx.var`.
2022-12-06 21:22:56,703 (client:306) INFO: {'Role': 'Client #1', 'Round': 2, 'Results_raw': {'train_acc': 0.364414, 'train_total': 1169, 'train_loss': 2097.855597, 'train_avg_loss': 1.794573}}
2022-12-06 21:23:00,110 (client:306) INFO: {'Role': 'Client #3', 'Round': 2, 'Results_raw': {'train_acc': 0.412195, 'train_total': 1640, 'train_loss': 2594.247381, 'train_avg_loss': 1.581858}}
2022-12-06 21:23:03,130 (client:306) INFO: {'Role': 'Client #2', 'Round': 2, 'Results_raw': {'train_acc': 0.267969, 'train_total': 1433, 'train_loss': 3005.513627, 'train_avg_loss': 2.097358}}
2022-12-06 21:23:03,243 (server:335) INFO: ----------- Starting a new training round (Round #3) -------------
2022-12-06 21:23:06,647 (client:306) INFO: {'Role': 'Client #3', 'Round': 3, 'Results_raw': {'train_acc': 0.473171, 'train_total': 1640, 'train_loss': 2338.653491, 'train_avg_loss': 1.426008}}
2022-12-06 21:23:09,206 (client:306) INFO: {'Role': 'Client #1', 'Round': 3, 'Results_raw': {'train_acc': 0.443969, 'train_total': 1169, 'train_loss': 1788.425374, 'train_avg_loss': 1.529876}}
2022-12-06 21:23:12,214 (client:306) INFO: {'Role': 'Client #2', 'Round': 3, 'Results_raw': {'train_acc': 0.300768, 'train_total': 1433, 'train_loss': 2753.047348, 'train_avg_loss': 1.921177}}
/home/jones.wz/anaconda3/envs/fs/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3440: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/jones.wz/anaconda3/envs/fs/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
2022-12-06 21:23:12,220 (server:593) INFO: {'Role': 'Server #', 'Round': 2, 'Results_weighted_avg': {'test_acc': 0.22716, 'test_total': 405.0, 'test_loss': 821.534889, 'test_avg_loss': 1.996196, 'val_acc': 0.216529, 'val_total': 201.666667, 'val_loss': 411.385751, 'val_
avg_loss': 2.002012}, 'Results_avg': {'test_acc': 0.231187, 'test_total': 405.0, 'test_loss': 808.45956, 'test_avg_loss': 2.000993, 'val_acc': 0.224764, 'val_total': 201.666667, 'val_loss': 403.73907, 'val_avg_loss': 1.999528}, 'Results_fairness': {'test_total': 405.0,
'val_total': 201.666667, 'test_acc_std': 0.029981, 'test_acc_bottom_decile': 0.202128, 'test_acc_top_decile': 0.272455, 'test_acc_min': 0.202128, 'test_acc_max': 0.272455, 'test_acc_bottom10%': nan, 'test_acc_top10%': 0.272455, 'test_acc_cos1': 0.991696, 'test_acc_entro
py': 1.090368, 'test_loss_std': 99.240095, 'test_loss_bottom_decile': 669.849205, 'test_loss_top_decile':




It seems that the evaluation is conducted at the end of round1. However, its results are printed after the display of round3's training results. This is likely to make users confused.


Discussion:

Resolving patch:
diff --git a/federatedscope/core/workers/server.py b/federatedscope/core/workers/server.py
--- a/federatedscope/core/workers/server.py
+++ b/federatedscope/core/workers/server.py
@@ -655,11 +655,14 @@ def broadcast_model_para(self,
         else:
             model_para = {} if skip_broadcast else self.model.state_dict()
 
+        # We define the evaluation happens at the end of an epoch
+        rnd = self.state - 1 if msg_type == 'evaluate' else self.state
+
         self.comm_manager.send(
             Message(msg_type=msg_type,
                     sender=self.ID,
                     receiver=receiver,
-                    state=min(self.state, self.total_round_num),
+                    state=min(rnd, self.total_round_num),
                     timestamp=self.cur_timestamp,
                     content=model_para))
         if self._cfg.federate.online_aggr:


Test patch: