Model not giving any output after full fine-tunining(Instruction based fine-tuning) on DDP

username: iammano
cooked: <p>Hey there below is my DDP Trainer code where everything is working all right the thing here is upon completion of training on the nodes participating in training. snapshot is saved and the saved snapshot is loaded for inference. Also, inference code is also provided below. Since it is instruction-based fine-tuning, I also mentioned the data encoding(tokenization) class for preprocessing the data provided end of this topic.</p>
<p>If you think the code is gibberish and not able to understand it easily, kindly reply to this topic I will respond ASAP.</p>
<p>My thinking:<br>
I’m thinking of something missing in the training step or doing the DDP training incorrectly.</p>
<p>Kindly take a look at it and help me out on this. Thank you very much.</p>
<h1><a name="cmd-used2-machines-node_rank-will-change-accordingly-1" class="anchor" href="#cmd-used2-machines-node_rank-will-change-accordingly-1"></a>Cmd used(2 machines node_rank will change accordingly):</h1>
<p><code>torchrun --nproc-per-node=1 --nnode=2 --node_rank=0 --rdzv_id=123 --rdzv_endpoint=hostname:12345 --rdzv_backend=c10d Trainer_CPU 5 1</code></p>
<h1><a name="trainer-code-2" class="anchor" href="#trainer-code-2"></a>Trainer Code:</h1>
<p>def ddp_setup():<br>
init_process_group(backend=“gloo”)</p>
<p>class Trainer:<br>
def <strong>init</strong>(<br>
self,<br>
model: torch.nn.Module,<br>
train_data: DataLoader,<br>
save_every: int,<br>
snapshot_path: str,<br>
optimizer<br>
) → None:<br>
self.local_rank = int(os.environ[“LOCAL_RANK”])<br>
self.global_rank = int(os.environ[“RANK”])<br>
self.model = model<br>
self.train_data = train_data<br>
self.optimizer = optimizer<br>
self.save_every = save_every<br>
self.epochs_run = 0<br>
self.snapshot_path = snapshot_path<br>
if os.path.exists(snapshot_path):<br>
print(“Loading snapshot”)<br>
self._load_snapshot(snapshot_path)<br>
self.model = DDP(self.model)</p>
<pre><code>def _load_snapshot(self, snapshot_path):
    snapshot = torch.load(snapshot_path, map_location=torch.device("cpu"))
    self.model.load_state_dict(snapshot["MODEL_STATE"])
    self.epochs_run = snapshot["EPOCHS_RUN"]
    print(f"Resuming training from snapshot at Epoch {self.epochs_run}")

def _save_snapshot(self, epoch):
    snapshot = {
        "MODEL_STATE": self.model.module.state_dict(),
        "EPOCHS_RUN": epoch,
    }
    torch.save(snapshot, self.snapshot_path)
    torch.save(self.model, "model.pt")
    print(f"Epoch {epoch+1} | Training snapshot saved at {self.snapshot_path}")

def _run_batch(self, source, targets):
    self.optimizer.zero_grad()
    output = self.model(source)
    loss = F.cross_entropy(output, targets)
    loss.backward()
    self.optimizer.step()
    
def _custom_run_batch(self, source):
    self.optimizer.zero_grad()
    output = self.model(**source, labels=source["input_ids"].to(torch.long))
    loss = output.loss
    loss.backward()     
    self.optimizer.step()
    print(f"CPU: {self.global_rank} | Loss: {loss} ") 

def _run_epoch(self, epoch):
    b_sz = len(next(iter(self.train_data))["input_ids"])
    print(f"[CPU{self.global_rank}] Epoch {epoch+1} | Batchsize: {b_sz} | Steps: {len(self.train_data)}")
    self.train_data.sampler.set_epoch(epoch)
    for batch in self.train_data:
        batch = {k:v.to(f"cpu:{self.local_rank}") for k, v in batch.items()}
        self._custom_run_batch(batch)

def train(self, max_epochs: int):
    for epoch in range(self.epochs_run, max_epochs):
        self._run_epoch(epoch)
        if (self.local_rank == 0 and epoch % self.save_every == 0) or (epoch == max_epochs-1):
            self._save_snapshot(epoch)
</code></pre>
<p><span class="hashtag-raw">#Need</span> check specific about dataset and dataloader<br>
def prepare_dataloader(dataset: Dataset, batch_size: int):<br>
return DataLoader(<br>
dataset,<br>
batch_size=batch_size,<br>
pin_memory=True,<br>
shuffle=False,<br>
sampler=DistributedSampler(dataset)<br>
)</p>
<p><span class="hashtag-raw">#GPT-Neo</span></p>
<p><span class="hashtag-raw">#Loading</span> the Dataset<br>
def load_data(tokenizer):<br>
df = pd.read_csv(“train_data.csv”)<br>
X_train,  y_train = df[“document”].tolist(), df[“summary”].tolist()<br>
train_dataset = Encoding(X_train, y_train, tokenizer, max_length=512)<br>
return train_dataset.data</p>
<p>def prepare_dataloader(dataset: list, batch_size: int):<br>
return DataLoader(<br>
dataset,<br>
batch_size=batch_size,<br>
pin_memory=True,<br>
shuffle=False,<br>
sampler=DistributedSampler(dataset)<br>
)</p>
<p>def llm_load_train_objs(batch_size):<br>
<span class="hashtag-raw">#Initialize</span> model<br>
model_name = ‘EleutherAI/gpt-neo-1.3B’<br>
tokenize = GPT2Tokenizer.from_pretrained(model_name, bos_token=‘’, eos_token=“”, pad_token=“”, max_length=512)<br>
model = GPTNeoForCausalLM.from_pretrained(model_name)<br>
model.resize_token_embeddings(len(tokenize))<br>
for buffers in model.buffers():<br>
if buffers.dtype == torch.bool:<br>
buffers.data = buffers.data.type(torch.uint8)<br>
torch.manual_seed(42)<br>
train_data = load_data(tokenize)<br>
dataset = prepare_dataloader(train_data, batch_size)<br>
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)<br>
return model, dataset, optimizer</p>
<p>def main(save_every: int, total_epochs: int, batch_size: int, snapshot_path: str = “snapshot.pt”):<br>
ddp_setup()<br>
model, dataset, optimizer = llm_load_train_objs(batch_size)<br>
print(len(dataset))<br>
trainer = Trainer(model, dataset, save_every, snapshot_path, optimizer)<br>
start_time = time.time()<br>
trainer.train(total_epochs)<br>
end_time = time.time()<br>
print("Total Time taken: “, end_time - start_time, " seconds”)<br>
destroy_process_group()</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
import argparse<br>
parser = argparse.ArgumentParser(description=‘simple distributed training job’)<br>
parser.add_argument(‘total_epochs’, type=int, help=‘Total epochs to train the model’)<br>
parser.add_argument(‘save_every’, type=int, help=‘How often to save a snapshot’)<br>
parser.add_argument(‘–batch_size’, default=2, type=int, help=‘Input batch size on each device (default: 32)’)<br>
args = parser.parse_args()<br>
main(args.save_every, args.total_epochs, args.batch_size)`</p>
<h1><a name="inference-codeusing-jupyter-notebook-here-3" class="anchor" href="#inference-codeusing-jupyter-notebook-here-3"></a>Inference code(using jupyter-notebook here):</h1>
<p>model_name = “EleutherAI/gpt-neo-1.3B”<br>
tokenize = GPT2Tokenizer.from_pretrained(model_name, bos_token=‘’, eos_token=“”, pad_token=“”, max_length=512)<br>
model = GPTNeoForCausalLM.from_pretrained(model_name)<br>
model.resize_token_embeddings(len(tokenize))<br>
for buffers in model.buffers():<br>
if buffers.dtype == torch.bool:<br>
buffers.data = buffers.data.type(torch.uint8)<br>
torch.manual_seed(42)<br>
snapshot_path = ‘snapshot.pt’<br>
snapshot = torch.load(snapshot_path, map_location=torch.device(‘cpu’))<br>
model.load_state_dict(snapshot[“MODEL_STATE”])</p>
<p>content = “toyota team europe were banned from the world rally championship.”<br>
instruct_prompt_2 = f’Summarize the given content:{content}\nSummary:’</p>
<p>generated = tokenize(f"{instruct_prompt_2}", return_tensors=‘pt’).input_ids</p>
<p>for buffers in model.buffers():<br>
if buffers.dtype == torch.uint8:<br>
buffers.data = buffers.data.type(torch.bool)</p>
<p>sample_outputs = model.generate(generated, top_k=50, max_length=200)</p>
<p>predicted_text = tokenize.decode(sample_outputs[0])<br>
print(predicted_text)</p>
<h1><a name="data-encoding-class-4" class="anchor" href="#data-encoding-class-4"></a>Data Encoding class:</h1>
<p>class Encoding():<br>
def <strong>init</strong>(self, content, summary, tokenizer, max_length):<br>
self.input_ids = []<br>
self.attn_masks = []<br>
self.labels = summary<br>
self.data = []</p>
<pre><code>    for content, summary in zip(content, summary):
        prep_txt = f'&lt;start&gt;content:{content}\nsummary:{summary}&lt;stop&gt;'
        #tokenize
        encodings_dict = tokenizer(prep_txt, truncation=True, max_length=max_length, padding="max_length")
        encodings_dict["input_ids"] = torch.tensor(encodings_dict["input_ids"]).to(torch.int32)
        encodings_dict["attention_mask"] = torch.tensor(encodings_dict["attention_mask"]).to(torch.int8)
        
        self.data.append(encodings_dict)

def __len__(self):
    return len(self.input_ids)

def __getitem__(self, idx):
    return self.input_ids[idx], self.attn_masks[idx], self.labels[idx]
</code></pre>
<p>Thanks,<br>
Mano</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="iammano" data-post="1" data-topic="185390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iammano/48/61700_2.png" class="avatar"> iammano:</div>
<blockquote>
<p>code is gibberish and not able to understand it easily, kindly reply to this topic I will respond ASAP.</p>
</blockquote>
</aside>
<p>Hi, Not sure if I fully understand what you are asking here.</p>
<ol>
<li>When you load the model from checkpoint before finetune, are you able to get output from the model?</li>
<li>After fine-tune, in which step did you not see output?</li>
</ol>

username: iammano
cooked: <p>Hi <a class="mention" href="/u/fduwjj">@fduwjj</a> Thanks for reading this topic.</p>
<p>What I’m trying?<br>
I’m trying to fine-tune a model for summarization using GPT-NEO-1.3B as the base model, upon using the HuggingFace Trainer class, the fine-tuning succeeds but trying to train with PyTorch-DDP for faster fine-tuning succeeds but the snapshot saved with DDP training is not giving any output it simply giving the pad token which I defined in the tokenizer.</p>
<p><strong>For eg:</strong>(Predicted by model)</p>
<p>Summarize the given content:<br>
australia 's current account deficit shrunk by a record 3.300 billion dollars billion us -rrb- in the june quarter due to soaring commodity prices , figures released monday showed .<br>
Summary:…</p>
<p>It is a sample example that I tried on the model after fine-tuning with DDP. Kindly help me with this.</p>
<p>I’m thinking of either the training step might be wrong, any resource suggestion will be great too.</p>

username: fduwjj
cooked: <p>Are you saying that after using DDP for fine-tuning, the model is not working properly now?</p>

username: iammano
cooked: <p>Exactly!, <a class="mention" href="/u/fduwjj">@fduwjj</a> model is not giving any response, it simply spits out what I’m giving as prompt.</p>

username: fduwjj
cooked: <p>OK, then this might be hard to debug… OK, one thing you can use here to get the <code>state_dict</code> of your model and see how it gets changed along the way. For example, when you loaded it from torch.load, and after DDP wrap, and after fine-tune before you save it to checkpoint, how does the weights/params change? Are these changes expected? I hope this can give you some glue on what and where things go wrong.</p>

username: iammano
cooked: <p>Hi <a class="mention" href="/u/fduwjj">@fduwjj</a> thanks for providing your thoughts, definitely will check and you are saying that after loading the base llm compare it’s weights with fine-tuned lllm, that’s what you are saying right?</p>

username: fduwjj
cooked: <p>I guess I was trying to suggest to keep track of the weight changes along the way (for every stage in your code, for example, loading, before fine-tune, after fine-tune, etc) Because DDP itself is not supposed to change the model and it lets the model training to like as if it happens on single machine.</p>
<p>So it might be because how you fine-tune it subject to batch size? If I understand it correctly, originally, you only use one GPU for fine tune and your batch size is b_1 and now using DDP, you keep the global batch size to be b_1 (batch size per GPU will be b_1 // # of GPUs) too right?</p>

username: iammano
cooked: <p>Hi <a class="mention" href="/u/fduwjj">@fduwjj</a> ,sorry for the delayed response.</p>
<p>What you are saying is correct, let me check with the changes of weights as you mentioned.I’ll give me a try!</p>
<p>Thanks</p>