Using Two Optimizers for Encoder and Decoder respectively vs using a single Optimizer for Both

username: kraken
cooked: <p>I have a theoretical doubt regarding the use of a different instance of optimizers for encoder and decoder respectively vs using a single optimizer for the complete autoencoder network.</p>
<p>I initially thought it was the same, but then I conducted an experiment on this and found, the loss of multiple optimizers tends to remain lower comparatively.</p>
<p>Code for the two versions is as follows:</p>
<pre><code class="lang-auto">def get_optimizer(config, net):

    if config.OPTIMIZER == 'Adam' or config.OPTIMIZER == 'adam':
        optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), \
                                   lr=config.BASE_LR, betas=(0.9, 0.999), 
                                   weight_decay=config.WEIGHT_DECAY)
    elif config.OPTIMIZER == 'SGD' or config.OPTIMIZER == 'sgd':
        optimizer = optim.SGD(net.parameters(), lr=config.BASE_LR, momentum=config.LEARNING_MOMENTUM, weight_decay=config.WEIGHT_DECAY)

    else:
        raise NotImplementedError
    
    return optimizer
</code></pre>
<p>For Multi Optimizers</p>
<pre><code class="lang-auto">def create_optimizers(config, enc, dec):
    net_encoder, net_decoder = enc, dec
    if config.OPTIMIZER == 'SGD' or config.OPTIMIZER == 'sgd':
        optimizer_encoder = optim.SGD(
            group_weight(net_encoder),
            lr=config.BASE_LR,
            momentum=config.LEARNING_MOMENTUM,
            weight_decay=config.WEIGHT_DECAY)
        optimizer_decoder = optim.SGD(
            group_weight(net_decoder),
            lr=config.BASE_LR,
            momentum=config.LEARNING_MOMENTUM,
            weight_decay=config.WEIGHT_DECAY)
    elif config.OPTIMIZER == 'Adam' or config.OPTIMIZER == 'adam':
        optimizer_encoder = optim.Adam(
            group_weight(net_encoder),
            lr=config.BASE_LR,
            betas=(0.9, 0.999),
            weight_decay=config.WEIGHT_DECAY)
        optimizer_decoder = optim.Adam(
            group_weight(net_decoder),
            lr=config.BASE_LR,
            betas=(0.9, 0.999),
            weight_decay=config.WEIGHT_DECAY)
    return (optimizer_encoder, optimizer_decoder)
</code></pre>
<p>The Loss curves are as follows<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/3972090a5adc0a6e9183df49cda411938e05f98d.png" data-download-href="https://discuss.pytorch.org/uploads/default/3972090a5adc0a6e9183df49cda411938e05f98d" title="Screenshot 2020-05-06 at 8.36.11 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/3972090a5adc0a6e9183df49cda411938e05f98d_2_521x500.png" alt="Screenshot 2020-05-06 at 8.36.11 PM" data-base62-sha1="8cbxbSNueywfG1c5qP1MLCwvwBf" width="521" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/3972090a5adc0a6e9183df49cda411938e05f98d_2_521x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/3972090a5adc0a6e9183df49cda411938e05f98d_2_781x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/3972090a5adc0a6e9183df49cda411938e05f98d.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-05-06 at 8.36.11 PM</span><span class="informations">942×904 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can somebody explain this or is this just a coincidence.</p>

username: JuanFMontesinos
cooked: <p>I Think it’s just a coincidence unless you have some bug , for example, you missed some parameter, you are wronly calling zero_grad and backward.</p>
<p>For example, in your code above you are only passing those parameters which are “trainable”, however gradients keep being accumulated as they won’t be affected by optim.zero_grad(). Parameters “after” those ones will be exposed to accumulated gradients. I doubt that’s intentional.</p>

username: kraken
cooked: <p>If a parameter is not trainable, will its accumulated gradient affect the training or testing phase in any manner ?</p>

username: JuanFMontesinos
cooked: <p>I would say it won’t affect for most of cases, unless you are working with techniques which involves gradients (like gradient alignment or so).</p>
<p>Can you check the ordering you are calling optimizers steps and zero grads? I think that’s probably the issue.</p>

username: kraken
cooked: <p>This is the Call</p>
<pre><code class="lang-auto">if self.config.MULTI_OPTIM:
					net.zero_grad()
					loss.backward()
					for opt in optimizer:
						clip_gradient(opt, self.config.GRADIENT_CLIP_NORM)
						opt.step()
				else:
					optimizer.zero_grad()
					loss.backward()
					clip_gradient(optimizer, self.config.GRADIENT_CLIP_NORM)
					optimizer.step()
</code></pre>
<p>I am training a simple autoencoder</p>

username: JuanFMontesinos
cooked: <aside class="quote no-group" data-username="kraken" data-post="5" data-topic="80019">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kraken/48/23849_2.png" class="avatar"> kraken:</div>
<blockquote>
<p>GRADIENT_CLIP_NORM</p>
</blockquote>
</aside>
<p>It’s a bit strange, it should be exactly the same (statistically talking).<br>
I would check <code>group_weight</code>, perhaps you can rewrite <code>create_optimizers</code> in function of <code>get_optimizer</code> to be sure you are doing the same in both cases.</p>
<p>Are you keeping the same validation set in both cases?</p>
<p>I mean, optimizer does nothing but iterating through parameters and setting new values.</p>
<pre><code class="lang-auto">    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            weight_decay = group['weight_decay']
            momentum = group['momentum']
            dampening = group['dampening']
            nesterov = group['nesterov']

            for p in group['params']:
                if p.grad is None:
                    continue
                d_p = p.grad
                if weight_decay != 0:
                    d_p = d_p.add(p, alpha=weight_decay)
                if momentum != 0:
                    param_state = self.state[p]
                    if 'momentum_buffer' not in param_state:
                        buf = param_state['momentum_buffer'] = torch.clone(d_p).detach()
                    else:
                        buf = param_state['momentum_buffer']
                        buf.mul_(momentum).add_(d_p, alpha=1 - dampening)
                    if nesterov:
                        d_p = d_p.add(buf, alpha=momentum)
                    else:
                        d_p = buf

                p.add_(d_p, alpha=-group['lr'])

        return loss
</code></pre>
<p>So as you can see it’s nothing but a for loop.</p>

username: cskarthik7
cooked: <p>If the optimizer uses other paramters values to calculate the gradient of one parameter then it will affect the training process.</p>