opened
lvxjl
username_0
which simulates the updated student weights via **gradient ascent** in order to take part in the calculation of teacher weights.

It turns out that updating student weights in "simulate_sgd_update" should be in accordance with updating student weights via SGD as in line 23. But in practice gradient **ascent** is used in one case while gradient **descent** is used in the other.

P.S. It also a little bit confusing to simulate the updated student weights right before you actually update them. Why not just take the updated weights from the student model? That's the **real** updated weights afterall.
?Error in simulate_sgd_update for MetaMatchingNetwork
issue
created
Z7zuqer
username_1
Hi,

Thanks for your interest in our project!

* We define function *simulate_sgd_update* to simulate gradients update of student weights. [Real SGD updates](https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD) utilize momentum and weight_decay to update student weights. In the updates of MetaMN, we only focus on the updates of the Meta Matchnig Networks, ignoring the updates of student weights. So we only need an approximation of updated student weights to see whether updated student weights are better than before to evaluate the performance of MetaMN. Both gradient ascent and descent are all allowed here.

```
# SGD update in Pytorch Official Implemention
if weight_decay != 0:
    d_p = d_p.add(p, alpha=weight_decay)
if momentum != 0:
    param_state = self.state[p]
    if 'momentum_buffer' not in param_state:
        buf = param_state['momentum_buffer'] = torch.clone(d_p).detach()
    else:
        buf = param_state['momentum_buffer']
        buf.mul_(momentum).add_(d_p, alpha=1 - dampening)
    if nesterov:
        d_p = d_p.add(buf, alpha=momentum)
    else:
        d_p = buf

```
* In the procedure of updating MetaMN, only weights of meta matching networks should be updated and student weights should be fixed. To make sure student weights are well stored in the process of updating MetaMN, we don't use optimizer.step() to directly update the student weights. This is to prevent `momentum_buffer` in SGD optimizer to be affected(If 'momentum_buffer' is affected, student weights may not correctly reach the global minimum). So here we define the function *simulate_sgd_update* to simulate the updates of student weights.

Best,
Hao.
comment
created
lvxjl
username_0
It turns out to be quite counterintuitive to see that using gradient ascent is OK.
And I am not sure if I fully understand how the performance of MetaMN depends on the weights of student network.
It would be great if you could provide more details, thank you.
comment
closed
penghouwen
username_2
issue