Why are the Values of nn.Parameter not Updating in PyTorch Training?

username: External_happy
cooked: <p>I’m trying to understand why my variable <code>x = nn.Parameter(torch.zeros((1, vision_embed_dim)))</code> doesn’t seem to update during training. I’ve set up an optimizer and scheduler, and everything appears to be functioning correctly. I’ve checked <code>x.grad</code>, and its values are updated every iteration. However, the values in the <code>x</code> tensor remain unchanged throughout each epoch. Why are the values in <code>x</code> not updating? I increase the learning rate and no update.</p>

username: agu
cooked: <p>Maybe you are not passing the parameter properly to the optimizer. It might be worth checking that / sharing your code.</p>

username: External_happy
cooked: <p>The centroid is not being updated or changed in the code.</p>
<pre><code class="lang-auto">class UniMoS:     
    def __init__(self, args, config) -&gt; None:
        super().__init__()
        ## -------------------------------------------------------
        self.args = args
        self.dataset_names = self.args.dataset
        self.class_num = args.class_num
        self.config = config
        self.logger = args.logger
        ## ------------ load data/ info --------------------------------
        self.source_loaders, self.target_loaders, self.val_loaders, self.classes_names = args.load_data_method(config)
        self.src_loader_len = len(self.source_loaders)
        self.tar_loader_len = len(self.target_loaders)
        ## ------------ load clip --------------------------------
        self.model = CustomModel(self.classes_names, 
                                self.dataset_names, 
                                self.args.target, 
                                self.args.source, 
                                self.target_loaders,
                                self.source_loaders)     
        self.model.cuda()
        ## -----------------------------------------------------------------------
        param_group = self.train_param()
        self.optimizer = torch.optim.SGD(param_group, momentum=0.9, weight_decay=0)
        self.epoch = -1
        self.num_batches = min(self.src_loader_len, self.tar_loader_len, 1500)
        ## -----------------------------------------------------------------------
        

    def train(self):
        train_loader_x = self.source_loaders
        train_loader_u = self.target_loaders
        for self.epoch in range(0, self.args.epochs):
            self.set_train()
            for i in tqdm(range(self.num_batches), mininterval=0.5):
                ## ---------------------------------------------------
                # idx_step = self.epoch * self.num_batches + i
                # self.inv_lr_scheduler(idx_step)
                ## ---------------------------------------------------
                train_loader_x_iter = iter(train_loader_x)
                train_loader_u_iter = iter(train_loader_u)
                batch_x = next(train_loader_x_iter, None)
                if batch_x is None:
                    train_loader_x_iter = iter(train_loader_x)
                    batch_x = next(train_loader_x_iter)
                batch_u = next(train_loader_u_iter, None)
                if batch_u is None:
                    train_loader_u_iter = iter(train_loader_u)
                    batch_u = next(train_loader_u_iter)
                ## ---------------------------------------------------
                images_x, labels_x = self.get_batch_data(batch_x)
                images_u, labels_u = self.get_batch_data(batch_u)
                with autocast():
                    ## ------------- CE on source -----------------------------
                    logits_x, img_fea_x, txt_feat_x = self.model(images_x, d="s")
                    loss = F.cross_entropy(logits_x, labels_x)
                    ## ------------- im loss on target ------------------------
                    logits_u, img_fea_u, txt_feat_u = self.model(images_u, d="t")
                    logits_proto = 100 * img_fea_t @ self.model.centroid_t.t()
                    loss += compute_im_loss(logits_proto)
                    self.optimizer.zero_grad()
                    loss.backward(retain_graph=True)
                    self.optimizer.step()
            self.test()

    def test(self):
        self.set_eval()
        total = 0
        correct = 0
        with torch.no_grad():
            for i, (images, target) in enumerate(tqdm(self.val_loaders)):
                with autocast(dtype=torch.float32):
                    images = images.cuda()
                    target = target.cuda()
                    total += target.shape[0]
                    output = self.model(images, d="t")[0]
                    correct += (torch.argmax(output, -1)==target).sum()
        acc = correct/total
        self.logger.info("Epoch {}: Acc={:.2f}%".format(self.epoch, acc * 100))
    
    def inv_lr_scheduler(self, iter_num, power=0.75, gamma=0.001):
        lr = self.args.lr * (1 + gamma * iter_num) ** (-power)
        for param_group in self.optimizer.param_groups:
            if param_group.get('unchange', True):
                param_group['lr'] = lr 
            else:
                # param_group['lr'] = self.args.lr * 0.1
                param_group['lr'] = self.args.lr * 0.1

    def train_param(self):
        print('Turning off gradients in both the image and the text encoder')
        print('-------------------------------------------------------------')
        learnable_params = []
        for name, param in self.model.named_parameters():
            param.requires_grad_(False)
            if "adapter" in name:
                param.requires_grad_(True)
            if "centroid_t" in name:
                param.requires_grad_(True)
            if param.requires_grad:
                print(f"params : {name}")
                learnable_params.append(param)
        print('-------------------------------------------------------------')
        param_group = [{'params': learnable_params, 'lr': self.args.lr}]
        return param_group
</code></pre>

username: KFrank
cooked: <p>Hi Trang!</p>
<aside class="quote no-group" data-username="External_happy" data-post="3" data-topic="207346" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/external_happy/48/66173_2.png" class="avatar"> External_happy:</div>
<blockquote>
<p>The centroid is not being updated or changed in the code.</p>
</blockquote>
</aside>
<p>You haven’t shown us where <code>centroid_t</code> is defined.  The details will<br>
matter.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        param_group = self.train_param()
        self.optimizer = torch.optim.SGD(param_group, momentum=0.9, weight_decay=0)
</code></pre>
</blockquote>
</aside>
<p>Print out <code>self.optimizer.param_groups[0]['params']</code> to check<br>
what’s in there.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                    self.optimizer.zero_grad()
                    loss.backward(retain_graph=True)
                    self.optimizer.step()
</code></pre>
</blockquote>
</aside>
<p>First, why do you have <code>retain_graph=True</code>?  It looks like a (mostly)<br>
harmless mistake.</p>
<p>Check that <code>self.centroid_t.grad</code> is not <code>None</code> and has a non-zero<br>
(and not excessively small) value just before <code>self.optimizer.step()</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def train_param(self):
        ...
        for name, param in self.model.named_parameters():
            ...
            if param.requires_grad:
                print(f"params : {name}")
</code></pre>
</blockquote>
</aside>
<p>Does this print out <code>centroid_t</code>, or is it missing somehow?</p>
<p>Best.</p>
<p>K. Frank</p>