Error: Predictions and/or references don't match the expected format

yam: <p>This is my code for fine tuning a ViT model for identifying soil moisture state:</p>
<p>from google.colab import drive<br>
drive.mount (‘/content/drive’)</p>
<p>!pip install datasets transformers accelerate torch torchvision scikit-learn matplotlib</p>
<p>import os<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
import pandas as pd<br>
import torch<br>
from torchvision import datasets, transforms<br>
from torch.utils.data import Dataset, DataLoader<br>
from PIL import Image</p>
<p>sm_img = ‘/content/drive/MyDrive/sm_classification_unsplit_multiclass_imgs-20240929T051421Z-001/sm_classification_unsplit_multiclass_imgs’</p>
<p>moisture = datasets.ImageFolder(root=sm_img)</p>
<h1><a name="p-456759-check-the-classes-and-the-number-of-images-1" class="anchor" href="#p-456759-check-the-classes-and-the-number-of-images-1"></a>Check the classes and the number of images</h1>
<p>print(“Classes:”, moisture.classes)<br>
print(“Number of images:”, len(moisture.imgs))</p>
<p>“”“# <strong>Split</strong>”“”</p>
<p>from sklearn.model_selection import train_test_split<br>
from torch.utils.data import random_split</p>
<p>labels = [label for _, label in moisture.imgs]</p>
<p>total_size = len(moisture)<br>
train_size = int(total_size * 0.7)  # 70%<br>
valid_size = int(total_size * 0.2)   # 20%<br>
test_size = total_size - train_size - valid_size  # 10%</p>
<p>train_indices, temp_indices = train_test_split(np.arange(total_size),<br>
test_size=(valid_size + test_size),<br>
random_state=42,<br>
stratify=labels)</p>
<p>valid_indices, test_indices = train_test_split(temp_indices,<br>
test_size=test_size/(valid_size + test_size),<br>
random_state=42,<br>
stratify=[labels[i] for i in temp_indices])</p>
<p>train_data = torch.utils.data.Subset(moisture, train_indices)<br>
valid_data = torch.utils.data.Subset(moisture, valid_indices)<br>
test_data = torch.utils.data.Subset(moisture, test_indices)</p>
<p>train_load = DataLoader(train_data, batch_size=32, shuffle=True)<br>
valid_load = DataLoader(valid_data, batch_size=32, shuffle=False)<br>
test_load = DataLoader(test_data, batch_size=32, shuffle=False)</p>
<p>train_percent = (train_size / total_size) * 100<br>
valid_percent = (valid_size / total_size) * 100<br>
test_percent = (test_size / total_size) * 100</p>
<p>print(f"Training set size: {train_size} ({train_percent:.2f}%)“)<br>
print(f"Validation set size: {valid_size} ({valid_percent:.2f}%)”)<br>
print(f"Test set size: {test_size} ({test_percent:.2f}%)")</p>
<p>train_ds, valid_ds, test_ds = random_split(moisture, [train_size, valid_size, test_size])</p>
<p>“”“# <strong>Pre-processing</strong>”“”</p>
<p>from transformers import ViTImageProcessor</p>
<p>pretrained_model = “google/vit-base-patch16-224-in21k”<br>
processor = ViTImageProcessor.from_pretrained(pretrained_model)</p>
<p>processor</p>
<p>from torchvision.transforms import (<br>
CenterCrop,<br>
Compose,<br>
Normalize,<br>
RandomHorizontalFlip,<br>
RandomResizedCrop,<br>
ToTensor,<br>
Resize,<br>
)</p>
<h1><a name="p-456759-define-transformations-for-training-validation-and-testing-2" class="anchor" href="#p-456759-define-transformations-for-training-validation-and-testing-2"></a>Define transformations for training, validation, and testing</h1>
<p>train_transforms = Compose([<br>
RandomResizedCrop(224),<br>
RandomHorizontalFlip(),<br>
ToTensor()<br>
])</p>
<p>val_transforms = Compose([<br>
Resize(224),<br>
CenterCrop(224),<br>
ToTensor()<br>
])</p>
<p>test_transforms = Compose([<br>
Resize(224),<br>
CenterCrop(224),<br>
ToTensor()<br>
])</p>
<h1><a name="p-456759-custom-dataset-class-3" class="anchor" href="#p-456759-custom-dataset-class-3"></a>Custom Dataset class</h1>
<p>class TransformedDataset(Dataset):<br>
def <strong>init</strong>(self, dataset, transform=None):<br>
self.dataset = dataset<br>
self.transform = transform</p>
<pre><code>def __len__(self):
    return len(self.dataset)

def __getitem__(self, idx):
    # Get the image and label from the dataset
    image, label = self.dataset[idx]

    # Check if the image is a NumPy array
    if isinstance(image, np.ndarray):
        image = Image.fromarray(image)  # Convert NumPy array to PIL Image IF NECESSARY

    # Apply transformation if available
    if self.transform:
        image = self.transform(image)

    return {"pixel_values": image, "label": label}
</code></pre>
<p>train_transformed_ds = TransformedDataset(train_ds, transform=train_transforms)<br>
valid_transformed_ds = TransformedDataset(valid_ds, transform=val_transforms)<br>
test_transformed_ds = TransformedDataset(test_ds, transform=test_transforms)</p>
<p><span class="hashtag-raw">#DataLoaders</span><br>
train_loader = DataLoader(train_transformed_ds, batch_size=32, shuffle=True)<br>
valid_loader = DataLoader(valid_transformed_ds, batch_size=32, shuffle=False)<br>
test_loader = DataLoader(test_transformed_ds, batch_size=32, shuffle=False)</p>
<p><span class="hashtag-raw">#check</span> if the loaders work properly<br>
dataIterator = iter(train_loader)<br>
batch = next(dataIterator)</p>
<h1><a name="p-456759-print-the-type-and-content-of-the-batch-to-understand-its-structure-4" class="anchor" href="#p-456759-print-the-type-and-content-of-the-batch-to-understand-its-structure-4"></a>Print the type and content of the batch to understand its structure</h1>
<p>print(type(batch))  # Check the type of the batch<br>
print(batch)        # Print the entire batch to see its contents</p>
<h1><a name="p-456759-pixel_values-and-label-are-keys-in-the-batch-dictionary-5" class="anchor" href="#p-456759-pixel_values-and-label-are-keys-in-the-batch-dictionary-5"></a>‘pixel_values’ and ‘label’ are keys in the batch dictionary</h1>
<p>if isinstance(batch, dict):<br>
# Check the min and max values of the pixel values in the batch to ensure they are in range<br>
pixel_values = batch[‘pixel_values’]<br>
print(“Min pixel value:”, pixel_values.min().item())<br>
print(“Max pixel value:”, pixel_values.max().item())</p>
<pre><code>print(batch['pixel_values'].shape)
print(batch['label'])
</code></pre>
<p>else:<br>
print(“Batch is not a dictionary. Please check the DataLoader implementation.”)</p>
<p>def collate_fn(batch):<br>
return {<br>
‘pixel_values’: torch.stack([x[‘pixel_values’] for x in batch]),<br>
‘labels’: torch.tensor([x[‘labels’] for x in batch])<br>
}</p>
<p>!pip install evaluate</p>
<p>from evaluate import load</p>
<h1><a name="p-456759-load-the-metrics-6" class="anchor" href="#p-456759-load-the-metrics-6"></a>Load the metrics</h1>
<p>accuracy_metric = load(“accuracy”)<br>
precision_metric = load(“precision”)<br>
recall_metric = load(“recall”)<br>
f1_metric = load(“f1”)</p>
<p>def compute_metrics(p):<br>
predictions, references = p</p>
<pre><code># Compute the individual metrics
accuracy = accuracy_metric.compute(predictions=predictions, references=references)
precision = precision_metric.compute(predictions=predictions, references=references, average='weighted')
recall = recall_metric.compute(predictions=predictions, references=references, average='weighted')
f1 = f1_metric.compute(predictions=predictions, references=references, average='weighted')

# Print the metrics
print(f"Accuracy: {accuracy['accuracy']:.4f}")
print(f"Precision: {precision['precision']:.4f}")
print(f"Recall: {recall['recall']:.4f}")
print(f"F1-score: {f1['f1']:.4f}")

# return the metrics as a dictionary
return {
    'accuracy': accuracy['accuracy'],
    'precision': precision['precision'],
    'recall': recall['recall'],
    'f1': f1['f1']
}
</code></pre>
<p>“”"# <strong>TRAIN <span class="hashtag-raw">#1</span></strong></p>
<p>Batch size: 16, 8, 4<br>
Learning rate: 1e-3, 9e-4<br>
Epochs: 1-20</p>
<ol>
<li>1-20 epoch, 1e-3 lr, no dropout</li>
<li>Epoch 1 to 10: lr=0.001, Epochs 11 to 20: lr=0.0009. In both cases, Attention dropout=0, MLP dropout=0</li>
<li>Epoch 1-10 (lr=0.001), 11-20 epoch (lr=0.0009)</li>
<li>Epoch 1 to 8: lr=0.001, Attention dropout=0, MLP dropout=0; Epochs 9 to 20: lr=0.0009, Attention dropout=0.2, MLP dropout=0.1<br>
“”"</li>
</ol>
<p>from transformers import ViTForImageClassification</p>
<h1><a name="p-456759-define-the-labels-and-their-corresponding-id2label-and-label2id-mappings-7" class="anchor" href="#p-456759-define-the-labels-and-their-corresponding-id2label-and-label2id-mappings-7"></a>Define the labels and their corresponding id2label and label2id mappings</h1>
<p>labels = [“dry”, “wet”]  # Example labels<br>
id2label = {i: label for i, label in enumerate(labels)}<br>
label2id = {label: i for i, label in enumerate(labels)}</p>
<h1><a name="p-456759-load-the-pre-trained-model-8" class="anchor" href="#p-456759-load-the-pre-trained-model-8"></a>Load the pre-trained model</h1>
<p>pretrained_model = “google/vit-base-patch16-224-in21k”  # Replace with your desired model<br>
model = ViTForImageClassification.from_pretrained(<br>
pretrained_model,<br>
num_labels=len(labels),<br>
id2label=id2label,<br>
label2id=label2id,<br>
ignore_mismatched_sizes=True<br>
)</p>
<h1><a name="p-456759-print-the-model-architecture-9" class="anchor" href="#p-456759-print-the-model-architecture-9"></a>Print the model architecture</h1>
<p>print(model)</p>
<p>from transformers import TrainingArguments</p>
<p>training_args = TrainingArguments(<br>
output_dir=“/content/drive/MyDrive/sm_classification_unsplit_multiclass_imgs-20240929T051421Z-001/sm_out”,<br>
per_device_train_batch_size=96,<br>
eval_strategy=“steps”,<br>
num_train_epochs=100,<br>
fp16=True,<br>
save_steps=100,<br>
eval_steps=100,<br>
logging_steps=10,<br>
learning_rate=5e-3,<br>
save_total_limit=2,<br>
remove_unused_columns=False,<br>
push_to_hub=False,<br>
report_to=‘tensorboard’,<br>
load_best_model_at_end=True,<br>
)</p>
<p>from transformers import Trainer<br>
from datasets import load_dataset, DatasetDict</p>
<p><span class="hashtag-raw">#moisture</span> is loaded as a DatasetDict<br>
moisture = DatasetDict({<br>
‘train’: train_transformed_ds,<br>
‘validation’: valid_transformed_ds<br>
})</p>
<h1><a name="p-456759-trainer-10" class="anchor" href="#p-456759-trainer-10"></a>Trainer</h1>
<p>trainer = Trainer(<br>
model=model,<br>
args=training_args,<br>
train_dataset=moisture[‘train’],  # string indexing<br>
eval_dataset=moisture[‘validation’],<br>
compute_metrics=compute_metrics,<br>
)</p>
<p>train_results = trainer.train()<br>
trainer.save_model()<br>
trainer.log_metrics(“train”, train_results.metrics)<br>
trainer.save_metrics(“train”, train_results.metrics)<br>
trainer.save_state()</p>
<p>but while training, this appears:</p>
<p>ValueError: Predictions and/or references don’t match the expected format.<br>
Expected format: {‘predictions’: Value(dtype=‘int32’, id=None), ‘references’: Value(dtype=‘int32’, id=None)},<br>
Input predictions: [[-0.7597656   0.80566406]<br>
[ 2.1445312  -2.140625  ]<br>
[ 0.69384766 -0.6352539 ]<br>
…<br>
[ 0.86621094 -0.8149414 ]<br>
[ 0.9838867  -0.9321289 ]<br>
[-0.04721069  0.1307373 ]],<br>
Input references: [0 0 0 1 0 1 1 1 1 0 1 0 0 0 0 1 0 0 0 1 1 1 0 0 0 0 1 0 0 1 1 0 1 0 0 1 1<br>
0 0 0 0 0 0 0 0 1 0 1 0 1 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0 0 0 0 1 0 0 1 1 0<br>
1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0 0 1 0 0 1 0 0 0 0<br>
0 0 0 0 0 0 1 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 1 0<br>
0 0 0 0 0 0 0 0 0 1 0 1 0 1 1 0 0 0 0 1 1 1 0 0 1 0 1 1 0 1 1 1 1 0 1 1 1<br>
0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 0 1 0 1 1 1 1 0 1 0 1 1 1 1 0 0 1<br>
0 0 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 1 1 0 1 0 0 0 0<br>
0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 1 1 0 0 0 1 1 0 1 0 1 1 0 1 0 1 0 0 0<br>
0 0 0 0 0 1 0 1 1 1 1 0 1 0 0 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 1 0 1 1 1 1 1<br>
1 1 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0 1 1 0 1 0 1 0 0 0 0 0 0 0<br>
0 1 1 1 0 1 0 0 0 0 1 1 0 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 0 0 0 0 0 1 1 0 1<br>
0 0 1 1 0 1 0 0 1 0 1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 0 1 1 0 1 0 0 0 1 1 0 0<br>
1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 1 0 0 1 1 0 1 0 1 0 0 0<br>
1 1 0 0 1 0 0 1 1 0 0 0 1 1 1 0 1 1 1 1 0 0 0 0 1 1 1 0 1 1 0 1 1 0 0 1 1<br>
1 0 0 0 0 1 0 1 1 1 1 1 1 0 1 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0<br>
0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 1 1 1 0 0 0 1 0 0 1 0 0 1 1 0 0 1 0 0 0 0 0<br>
0]</p>
<p>I am pretty sure the input is already normalized. So I don’t know what happened here</p>