The model trained in PyTorch produces inconsistent predictions for the same image when processed individually versus in a batch.

Dishant_Dua: <p>I am noticing a significant difference in model predictions when running predictions on a <strong>single image</strong> versus the <strong>whole dataset</strong>. The model, which was trained using <strong>PyTorch</strong>, gives drastically different predictions for the same image when processed individually versus in a batch. Is there any way to ensure that the predictions are consistent for the same image when processed individually and in a batch?</p>

ptrblck: <p>How large are the relative and absolute errors?</p>

Dishant_Dua: <p>For single image its [0.37732467 0.2642143  0.35846105]<br>
and for that same image in batch its [0.3185594  0.40971586 0.2717247 ].</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing these differences?</p>

Dishant_Dua: <p>I am sharing the following code for training and predicting with my model. This is how the training process works:</p>
<ol>
<li>Training: The model is trained using the <code>Trainer</code> class from HuggingFace’s <code>transformers</code> library. The training arguments are set using <code>TrainingArguments</code>, and the model is trained on the <code>train_dataset</code> and evaluated on the <code>val_dataset</code>. After training, the model is saved.</li>
<li>Prediction: Once the model is trained, I load the saved model and training arguments. Then, I use the <code>Trainer</code> class again to predict on the <code>test_dataset</code>.</li>
</ol>
<p>Here’s the issue I’m encountering:</p>
<p>When I comment out the line <code>training_args_loaded.per_device_eval_batch_size = 1</code>, the evaluation batch size defaults to <code>320</code>. However, when I uncomment this line, the batch size is set to <code>1</code>. This change in batch size leads to a noticeable difference in the predictions for the same image—predicting with batch size <code>1</code> produces different results than when using batch size <code>320</code>.</p>
<p>Could you help me understand why this discrepancy occurs and suggest how I can ensure consistent predictions regardless of the batch size setting?</p>
<p>Below is the code:</p>
<p>from transformers import Trainer, TrainingArguments<br>
import torch<br>
from torch.utils.data import Dataset<br>
from transformers import PreTrainedModel, PretrainedConfig, TrainerCallback<br>
import torch<br>
import torch.nn.functional as F<br>
import numpy as np</p>
<p>numOfFeatures=128<br>
class SequenceDataset(Dataset):<br>
def <strong>init</strong>(self, X, y):<br>
self.X = torch.tensor(X, dtype=torch.float32)<br>
self.y = torch.tensor(y, dtype=torch.long)</p>
<pre><code>def __len__(self):
    return len(self.y)

def __getitem__(self, idx):
    return {"input_ids": self.X[idx], "labels": self.y[idx]}
</code></pre>
<p>class SequenceConfig(PretrainedConfig):<br>
model_type = “sequence_transformer”<br>
def <strong>init</strong>(self, num_features=numOfFeatures, num_classes=3, d_model=1024, nhead=4, num_layers=4, dim_feedforward=512, **kwargs):<br>
self.num_features = num_features<br>
self.num_classes = num_classes<br>
self.d_model = d_model<br>
self.nhead = nhead<br>
self.num_layers = num_layers<br>
self.dim_feedforward = dim_feedforward<br>
super().<strong>init</strong>(**kwargs)</p>
<p>class SequenceTransformer(PreTrainedModel):<br>
config_class = SequenceConfig</p>
<pre><code>def __init__(self, config):
    super().__init__(config)
    self.embedding = torch.nn.Linear(config.num_features, config.d_model)
    self.positional_encoding = torch.nn.Parameter(torch.zeros(1, config.d_model))
    encoder_layer = torch.nn.TransformerEncoderLayer(d_model=config.d_model, nhead=config.nhead, dim_feedforward=config.dim_feedforward, batch_first=True)
    self.transformer_encoder = torch.nn.TransformerEncoder(encoder_layer, num_layers=config.num_layers)
    self.fc = torch.nn.Linear(config.d_model, config.num_classes)

def forward(self, input_ids, labels=None):
   
    src = self.embedding(input_ids) + self.positional_encoding
    output = self.transformer_encoder(src)
    logits = self.fc(output)
    probs = F.softmax(logits, dim=-1)

    loss = None
    if labels is not None:
        loss_fct = torch.nn.CrossEntropyLoss()
        loss = loss_fct(logits, labels)
        
    if labels is not None:
        loss_fct = torch.nn.CrossEntropyLoss()
        loss = loss_fct(logits, labels)
        
    return {"loss": loss, "logits": logits, "probs": probs} if loss is not None else logits
</code></pre>
<h1><a name="p-461564-training-code-1" class="anchor" href="#p-461564-training-code-1"></a>Training Code</h1>
<pre><code>config = SequenceConfig()
model = SequenceTransformer(config)
metrics=[]

# Training Arguments
batchSize=32
numWarmUpSteps=int(np.shape(train_image)[0]/batchSize/numOfBreakpointsPerEpoch/10)
training_args = TrainingArguments(
    output_dir=path,
    num_train_epochs=1, 
    per_device_train_batch_size=batchSize,
    per_device_eval_batch_size=320,
    warmup_steps=numWarmUpSteps,
    weight_decay=0.1,
    logging_strategy='no',
    eval_strategy="epoch",
    save_strategy="epoch",
    metric_for_best_model="accuracy",
    save_only_model=True,
)

trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=val_dataset,
        compute_metrics=compute_metrics,
        callbacks =[]
)

train_output = trainer.train()
</code></pre>
<h1><a name="p-461564-prediction-code-2" class="anchor" href="#p-461564-prediction-code-2"></a>Prediction code</h1>
<p>training_args_loaded = torch.load(path+“\SavedModels\training_args.bin”)<br>
model_save_path = path+“\SavedModels\”<br>
model = load_model(model_save_path, SequenceTransformer)</p>
<h1><a name="p-461564-training_args_loadedper_device_eval_batch_size1-3" class="anchor" href="#p-461564-training_args_loadedper_device_eval_batch_size1-3"></a>training_args_loaded.per_device_eval_batch_size=1</h1>
<pre><code>    trainer = Trainer(model=model,compute_metrics=compute_metrics,args=training_args_loaded)
</code></pre>
<p>testPredictions=trainer.predict(torch.tensor(test_dataset, dtype=torch.float32))</p>