PyTorch Model Training - operands could not be broadcast together with shapes (1024,1024,5) (3,)

username: pchi
cooked: <p>Hey guys, I’m facing a problem trying to train a segmentation model, as I’m new with PyTorch.</p>
<p>I’m trying to reproduce code from Segmentation Models library and more specificaly from this <a href="https://github.com/qubvel/segmentation_models.pytorch/blob/master/examples/cars%20segmentation%20(camvid).ipynb" rel="noopener nofollow ugc">example code</a>, with a custom dataset.</p>
<p>The dataset contains photos of plants taken from different perpectives different days that either have a disease on their leaves or not. If a leaf contains a disease, then its mask contains the segmentation of the whole leaf. The photographs of the dataset were taken using multispectral imaging to capture the disease spectrum response at 460, 540, 640, 640, 700, 775 and 875 nm and are 1900x3000. So I want to have <em>input_channels=5</em> and the <em>mask classes are 6</em>.</p>
<p>So for example the training folder format of the dataset is:</p>
<pre><code class="lang-auto">    .
    ├── train_images
    │   ├── plant1_day0_pov1_disease
    │       ├── image460.jpg
    │       ├── image540.jpg
    │       ├── image640.jpg
    │       ├── image775.jpg
    │       ├── image875.jpg
    │   └── plant1_day0_pov2_disease
    │       ├── image460.jpg
    │       ├── image540.jpg
    │       ├── image640.jpg
    │       ├── image775.jpg
    │       ├── image875.jpg
    │   └── etc...
    ├── train_annot
    │   ├── plant1_day0_pov1_disease.png
    │   ├── plant1_day0_pov2_disease.png
    │   └── etc...
    etc...
</code></pre>
<p>I have made changes to the whole code in order to make it custom for this dataset (DataLoaders, augmentations, transformations into 1024x1024) and to make the model accept 5 channels as input. The problem is that when trying to do <em>train_epoch.run(train_loader)</em> <strong>I get a ValueError: operands could not be broadcast together with shapes (1024,1024,5) (3,)</strong>.</p>
<p>My code is available on <a href="https://colab.research.google.com/drive/1MLTjTSM62G2aJt5TJSHuD31j4nJf-ppE?usp=sharing" rel="noopener nofollow ugc">Colab here</a>. If you want to give you a sample of the dataset in order to reproduce it please feel free to ask me.</p>
<p>I would appreciate it if anyone could help me.<br>
Thanks in advance!</p>

username: ptrblck
cooked: <p>Based on the stacktrace of the error message:</p>
<pre><code class="lang-python">/usr/local/lib/python3.10/dist-packages/segmentation_models_pytorch/encoders/_preprocessing.py in preprocess_input(x, mean, std, input_space, input_range, **kwargs)
     13     if mean is not None:
     14         mean = np.array(mean)
---&gt; 15         x = x - mean
     16 
     17     if std is not None:

ValueError: operands could not be broadcast together with shapes (1024,1024,5) (3,) 
</code></pre>
<p>it seems your model normalizes the inputs with 3 values by default, so either disable it somehow or pass 5 values to <code>mean</code> (and most likely <code>std</code>).</p>

username: pchi
cooked: <p>Thanks for the fast response!<br>
So I tried changing the <strong>get_preprocessing</strong> into:</p>
<pre><code class="lang-auto">def get_preprocessing(preprocessing_fn, mean=None, std=None):
    transform = [
        A.Lambda(name='preprocessing_fn', image=preprocessing_fn),
        A.Lambda(name='to_tensor', image=to_tensor, mask=to_tensor),
    ]
    
    # If mean and std are provided, use them for normalization
    if mean is not None and std is not None:
        normalize = A.Normalize(mean=mean, std=std)
        transform.insert(1, normalize)
    
    return A.Compose(transform, is_check_shapes=False)
</code></pre>
<p>and then tried to define test mean and std values</p>
<pre><code class="lang-auto"># Define the mean and std values for your 5-channel data
mean_values = [5, 5, 5, 5, 5]
std_values = [5, 5, 5, 5, 5]

# Create preprocessing functions with the mean and std values
train_preprocessing = get_preprocessing(preprocessing_fn, mean=mean_values, std=std_values)
</code></pre>
<p>but nothing changes, same error mesage. Any ideas how to disable the normalization of the inputs?</p>

username: ptrblck
cooked: <p>If the stacktrace still points to the same line of code, add debug <code>print</code> statements and check the shape of all used objects to make sure the right stats are passed and used.</p>

username: pchi
cooked: <p>The problem is <a href="https://github.com/qubvel/segmentation_models.pytorch/blob/6db76a1106426ac5b55f39fba68168f3bccae7f8/segmentation_models_pytorch/encoders/resnet.py#L115C1-L123C10" rel="noopener nofollow ugc">here</a>:</p>
<pre><code class="lang-auto">for source_name, source_url in sources.items():
pretrained_settings[model_name][source_name] = {
    "url": source_url,
    "input_size": [3, 224, 224],
    "input_range": [0, 1],
    "mean": [0.485, 0.456, 0.406],
    "std": [0.229, 0.224, 0.225],
    "num_classes": 1000,
}
</code></pre>
<p>Do you know how I should I change those? I mean I could change the input_size and num_classes by hand but mean/stds etc?</p>

username: ptrblck
cooked: <p>You could calculate the stats from your training dataset and also change it manually. The posted values are from ImageNet and since your images contain captures spectral data the stats could also differ.</p>