Transformer translation task: interaction between norm bias and attention bias causing divergence

username: mrich
cooked: <p>Hello.</p>
<p>I’m training a transformer to replicate “attention is all you need” for german to english translation and I’ve found that training starts to diverge pretty quickly. I wanted to start with a baseline, so I was following the pytorch <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">transformer tutorial</a>. I.e. the architecture is 2 layers of an encoder, and then the decoder is just a linear layer and a softmax.</p>
<p>Here is an abridged version of the architecture:</p>
<pre><code class="lang-auto">class PositionalEncoding(nn.Module):
    def __init__(self, d_model: int, dropout: float = 0.1, max_len: int = 5000):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
        pe = torch.zeros(max_len, 1, d_model, device=device)
        pe[:, 0, 0::2] = torch.sin(position * div_term)
        pe[:, 0, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

    def forward(self, x: Tensor) -&gt; Tensor:
        """
        Args:
            x: Tensor, shape [seq_len, batch_size, embedding_dim]
        """
        # Logic above assumes [seq_len, batch_size, embedding_dim]
        # Swap axes for batch_size first, then swap back. 
        x = torch.swapaxes(x, 1, 0)
        x = x + self.pe[:x.size(0)]
        x = torch.swapaxes(x, 0, 1)        
        return self.dropout(x)  
    
    
class BaselineTransformer(nn.Module): 
    def __init__(self, 
                 model_dim=512, 
                 nhead=8, 
                 num_encoder_layers=6,
                 num_decoder_layers=6, 
                 output_vocab_size=10000, 
                 padding_idx=0, 
                 dropout=0.0): 
        super(BaselineTransformer, self).__init__()
        self.model_dimension = model_dim
        self.nhead = nhead
        self.num_encoder_layers = num_encoder_layers
        self.num_decoder_layers = num_decoder_layers
        self.output_vocab_size = output_vocab_size
        
        pretrained_model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-de-en")
        
        self.encoder_input_embeddings = pretrained_model.get_encoder().embed_tokens.to(device)
        
        self.pos_encoder = PositionalEncoding(model_dim, dropout=dropout)
        encoder_layers = TransformerEncoderLayer(d_model=model_dim, nhead=nhead, batch_first=True, dropout=dropout, norm_first=config['norm_first'])
        self.transformer_encoder = TransformerEncoder(encoder_layers, num_encoder_layers).to(device)#, encoder_norm).to(device)
    
        self.target_lm_head = nn.Linear(self.model_dimension, self.output_vocab_size, device=device)
        self.log_softmax = torch.nn.LogSoftmax(dim=2)   
        self.init_weights()
        
    def init_weights(self):
        initrange = 0.1
        self.encoder_input_embeddings.weight.data.uniform_(-initrange, initrange)
        self.target_lm_head.weight.data.uniform_(-initrange, initrange)
        self.target_lm_head.bias.data.uniform_(math.log(1 / self.output_vocab_size) - initrange, math.log(1 / self.output_vocab_size) + initrange)
        
    def forward(self, source, target, tokenizer, 
                src_key_padding_mask=None, tgt_key_padding_mask=None, tgt_mask=None, src_mask=None):
        source_embeddings_raw = self.encoder_input_embeddings(source) * math.sqrt(self.model_dimension)
        source_embeddings = self.pos_encoder(source_embeddings_raw)        
        memory = self.transformer_encoder(
            source_embeddings, 
            src_key_padding_mask=src_key_padding_mask,
            mask=src_mask
        ) 
        
        logits = self.target_lm_head(memory)
        res = self.log_softmax(logits)   
        return res
</code></pre>
<p>It looks like training loss reaches some low level (≈ 7.5 for NLL) and starts to diverge rapidly thereafter. It also appears as if there is some interaction between the bias of the layer norm and the bias of the attention operation.</p>
<p>Here is the loss: <a href="https://wandb.ai/markrich/vanilla-transformer-baseline-01-21-2022/reports/Shared-panel-22-01-30-13-01-06--VmlldzoxNTAyNjc2?accessToken=327e04f0q0kxuxzbl3x8fko7hb3p2920cbsm1hvceffhnneg119k4yorimia2bdi" rel="noopener nofollow ugc">link</a><br>
Image:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/051c5f8845e9f6401abd7ebce85865734fa1f116.png" data-download-href="https://discuss.pytorch.org/uploads/default/051c5f8845e9f6401abd7ebce85865734fa1f116" title="Screen Shot 2022-01-30 at 1.46.28 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/051c5f8845e9f6401abd7ebce85865734fa1f116_2_690x362.png" alt="Screen Shot 2022-01-30 at 1.46.28 PM" data-base62-sha1="JdaWPBdEVuRNh216VzY8CNg7lk" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/051c5f8845e9f6401abd7ebce85865734fa1f116_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/051c5f8845e9f6401abd7ebce85865734fa1f116_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/051c5f8845e9f6401abd7ebce85865734fa1f116.png 2x" data-dominant-color="FBFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-01-30 at 1.46.28 PM</span><span class="informations">1232×648 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here are some charts plotting the parameter values evolving over training:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f37eece291f90a243e8200db459f1e9e1163fa98.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f37eece291f90a243e8200db459f1e9e1163fa98" title="Group 1 (45)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f37eece291f90a243e8200db459f1e9e1163fa98_2_336x500.jpeg" alt="Group 1 (45)" data-base62-sha1="yK3XClCJ3lejLAEXx0JUha19bHW" width="336" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f37eece291f90a243e8200db459f1e9e1163fa98_2_336x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f37eece291f90a243e8200db459f1e9e1163fa98_2_504x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f37eece291f90a243e8200db459f1e9e1163fa98_2_672x1000.jpeg 2x" data-dominant-color="D4E5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Group 1 (45)</span><span class="informations">1851×2751 519 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here are some annotations of the chart that indicate an interaction between norm bias and attention bias (though not positive!)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82c2e83c3dceb47453e74a1d16bed78082aa9191.png" data-download-href="https://discuss.pytorch.org/uploads/default/82c2e83c3dceb47453e74a1d16bed78082aa9191" title="Group 2 (47)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c2e83c3dceb47453e74a1d16bed78082aa9191_2_336x500.png" alt="Group 2 (47)" data-base62-sha1="iELEMKi9G4ZlTxtya0BtmJBR4Ln" width="336" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c2e83c3dceb47453e74a1d16bed78082aa9191_2_336x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c2e83c3dceb47453e74a1d16bed78082aa9191_2_504x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c2e83c3dceb47453e74a1d16bed78082aa9191_2_672x1000.png 2x" data-dominant-color="D4E2F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Group 2 (47)</span><span class="informations">1851×2751 748 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m also using 1/5 the learning rate of the “attention is all you need” paper, so a bit surprised about the training divergence.</p>
<p>Finally, right around when the divergence starts happening, the attention weights of the first training layer get nuked, and it seems like it just starts paying attention to the same tokens for every other token (note the rows sum to 1 in this heatmap):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d84f8951b36015b392879e472b6c56f15918e19.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2d84f8951b36015b392879e472b6c56f15918e19" title="Screen Shot 2022-01-30 at 1.58.52 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d84f8951b36015b392879e472b6c56f15918e19_2_506x500.jpeg" alt="Screen Shot 2022-01-30 at 1.58.52 PM" data-base62-sha1="6uGngzckUNABccjLa2dQNWemHGh" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d84f8951b36015b392879e472b6c56f15918e19_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d84f8951b36015b392879e472b6c56f15918e19_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d84f8951b36015b392879e472b6c56f15918e19_2_1012x1000.jpeg 2x" data-dominant-color="636195"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-01-30 at 1.58.52 PM</span><span class="informations">1626×1606 228 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9a236b2d3d17f9db44fbc5fded157e4a0cd35c2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a9a236b2d3d17f9db44fbc5fded157e4a0cd35c2" title="Screen Shot 2022-01-30 at 1.59.08 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9a236b2d3d17f9db44fbc5fded157e4a0cd35c2_2_487x499.jpeg" alt="Screen Shot 2022-01-30 at 1.59.08 PM" data-base62-sha1="ocEeuharjftwEF3v06ysqZqC15U" width="487" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9a236b2d3d17f9db44fbc5fded157e4a0cd35c2_2_487x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9a236b2d3d17f9db44fbc5fded157e4a0cd35c2_2_730x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9a236b2d3d17f9db44fbc5fded157e4a0cd35c2_2_974x998.jpeg 2x" data-dominant-color="6A417C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-01-30 at 1.59.08 PM</span><span class="informations">1626×1666 174 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca414e42cb22d92c3eea09be1bbb0bb8c04ac7d9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ca414e42cb22d92c3eea09be1bbb0bb8c04ac7d9" title="Screen Shot 2022-01-30 at 1.59.21 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca414e42cb22d92c3eea09be1bbb0bb8c04ac7d9_2_517x499.jpeg" alt="Screen Shot 2022-01-30 at 1.59.21 PM" data-base62-sha1="sRemGn9Oq12YA8T9BZtjUc45DSh" width="517" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca414e42cb22d92c3eea09be1bbb0bb8c04ac7d9_2_517x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca414e42cb22d92c3eea09be1bbb0bb8c04ac7d9_2_775x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca414e42cb22d92c3eea09be1bbb0bb8c04ac7d9_2_1034x998.jpeg 2x" data-dominant-color="68417C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-01-30 at 1.59.21 PM</span><span class="informations">1670×1614 183 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any idea on what might be happening? This model doesn’t perform much better than an input-independent baseline (which reaches 8.5 NLL whereas this model hits around 7 before diverging).</p>
<p>Also, here’s the hyper parameters used for training in case that helps.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa3850f15ed2e399b8826416716ee889b85c5c64.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa3850f15ed2e399b8826416716ee889b85c5c64" title="Screen Shot 2022-01-30 at 2.02.49 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa3850f15ed2e399b8826416716ee889b85c5c64_2_690x255.png" alt="Screen Shot 2022-01-30 at 2.02.49 PM" data-base62-sha1="zHy1w5LYr98snQLfs3dFlVyNfJa" width="690" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa3850f15ed2e399b8826416716ee889b85c5c64_2_690x255.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa3850f15ed2e399b8826416716ee889b85c5c64_2_1035x382.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa3850f15ed2e399b8826416716ee889b85c5c64_2_1380x510.png 2x" data-dominant-color="060809"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-01-30 at 2.02.49 PM</span><span class="informations">1514×560 76.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/mrich">@mrich</a> This can happen with any NLL based loss optimizer.</p>
<ul>
<li>
<p>You can introduce smart learning_rate reduction when you find that the loss is not reducing for 2 to 3 epochs, otherwise the graph can pick up some incorrect gradient and move in that direction</p>
</li>
<li>
<p>You can reduce the batch_size to 16 or 32 as this allows the model to train in a more generic direction rather that specific after every epoch</p>
</li>
<li>
<p>You can switch over from advanced optimizers like Adam etc to SGD after a certain reduction of loss so that the optimizer efficiently manoeuvres the gradient drops</p>
</li>
</ul>
<p>This exercise is more of an art than science so you will only get tips. Good luck</p>

username: mrich
cooked: <p>Thank you <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a>!!</p>
<p>Just wanted to follow up here; I’ve only tried your second suggestion so far (changing batch size to 32), but it seems to lead to much better results. Really do appreciate your help; was stuck on this for a little while! Probably should’ve tried some sort of sweep over hyperparameters. Anyway, I’ve only ran this for a few hours so I’m sure it will get much better results later.</p>
<p>For anyone looking at this in the future:</p>
<p><strong>Losses / Validation Losses</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46f4757e89a6ac212faa0b7c9b37ef3cc91832f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/46f4757e89a6ac212faa0b7c9b37ef3cc91832f4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46f4757e89a6ac212faa0b7c9b37ef3cc91832f4_2_690x225.png" alt="image" data-base62-sha1="a7HaMfrQzkN8ykSAtpYgIq3dcGg" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46f4757e89a6ac212faa0b7c9b37ef3cc91832f4_2_690x225.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46f4757e89a6ac212faa0b7c9b37ef3cc91832f4_2_1035x337.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46f4757e89a6ac212faa0b7c9b37ef3cc91832f4_2_1380x450.png 2x" data-dominant-color="F4F8FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3700×1210 337 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Histogram of values for intermediate values</strong> (memory = encoder output, predicted embedding = decoder output)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ccd7a6552a40b51d0ff9bef4c26ca6b1b913a91.png" data-download-href="https://discuss.pytorch.org/uploads/default/1ccd7a6552a40b51d0ff9bef4c26ca6b1b913a91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ccd7a6552a40b51d0ff9bef4c26ca6b1b913a91_2_690x220.png" alt="image" data-base62-sha1="46NBrJGF1x2ZuNuYp5Yqzez35i9" width="690" height="220" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ccd7a6552a40b51d0ff9bef4c26ca6b1b913a91_2_690x220.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ccd7a6552a40b51d0ff9bef4c26ca6b1b913a91_2_1035x330.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ccd7a6552a40b51d0ff9bef4c26ca6b1b913a91_2_1380x440.png 2x" data-dominant-color="EEF3F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3732×1194 215 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Last Attention Layers</strong><br>
Screen Shot 2022-02-02 at 8.34.41 AM<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96f773bce5fd47a9f8d8d9da33154870e69f2f76.png" data-download-href="https://discuss.pytorch.org/uploads/default/96f773bce5fd47a9f8d8d9da33154870e69f2f76" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f773bce5fd47a9f8d8d9da33154870e69f2f76_2_514x500.png" alt="image" data-base62-sha1="lxvMCEteW81ba95TccPpZiG2qkC" width="514" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f773bce5fd47a9f8d8d9da33154870e69f2f76_2_514x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f773bce5fd47a9f8d8d9da33154870e69f2f76_2_771x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f773bce5fd47a9f8d8d9da33154870e69f2f76_2_1028x1000.png 2x" data-dominant-color="6B3F7E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1664×1618 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Screen Shot 2022-02-02 at 8.35.04 AM<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b088fc07e98de4d46cb8b373155c15541ea48036.png" data-download-href="https://discuss.pytorch.org/uploads/default/b088fc07e98de4d46cb8b373155c15541ea48036" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b088fc07e98de4d46cb8b373155c15541ea48036_2_534x500.png" alt="image" data-base62-sha1="pbHwpmCtNwJ3LLfiKaWh1NYzO8S" width="534" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b088fc07e98de4d46cb8b373155c15541ea48036_2_534x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b088fc07e98de4d46cb8b373155c15541ea48036_2_801x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b088fc07e98de4d46cb8b373155c15541ea48036_2_1068x1000.png 2x" data-dominant-color="734985"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1758×1644 230 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Will follow up with a more detailed write up via a blog post s.t. others can replicate later. Appreciate your help again.</p>