DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=18432, row_off=0, width=2048, height=2048)
DEBUG:rasterio._io:IO window xoff=18432.0 yoff=0.0 width=2048.0 height=2048.0
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x349798ca0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x3497655e0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x3497655e0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x349798ca0>
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
UpsampleBlock output shape: torch.Size([2, 256, 512, 512])
x.shape after upsampling: torch.Size([2, 256, 512, 512])
Encoder Feature 2 shape: torch.Size([2, 48, 512, 512])
x.shape after concatenation: torch.Size([2, 304, 512, 512])
Passing through UNetBlock with in_channels=304
x.shape after UNetBlock: torch.Size([2, 256, 512, 512])

Decoder Stage 3
x.shape before upsampling: torch.Size([2, 256, 512, 512])
UpsampleBlock input shape: torch.Size([2, 256, 512, 512])
UpsampleBlock output shape: torch.Size([2, 128, 1024, 1024])
x.shape after upsampling: torch.Size([2, 128, 1024, 1024])
Encoder Feature 3 shape: torch.Size([2, 24, 1024, 1024])
x.shape after concatenation: torch.Size([2, 152, 1024, 1024])
Passing through UNetBlock with in_channels=152
x.shape after UNetBlock: torch.Size([2, 128, 1024, 1024])

Decoder Stage 4
x.shape before upsampling: torch.Size([2, 128, 1024, 1024])
UpsampleBlock input shape: torch.Size([2, 128, 1024, 1024])
UpsampleBlock output shape: torch.Size([2, 64, 2048, 2048])
x.shape after upsampling: torch.Size([2, 64, 2048, 2048])
Passing through UNetBlock with in_channels=64
x.shape after UNetBlock: torch.Size([2, 64, 2048, 2048])
[2024-10-08 13:22:17,964][__main__][ERROR] - Error in batch 0: Sizes of tensors must match except in dimension 1. Expected size 256 but got size 128 for tensor number 5 in the list.
[W 2024-10-08 13:22:27,980] Trial 0 failed with parameters: {'num_layers': 4, 'num_filters': 64, 'kernel_size': 5, 'activation': 'relu', 'dropout_rate': 0.05885975030768492, 'learning_rate': 0.00021698295108635467, 'optimizer': 'SGD'} because of the following error: RuntimeError('Sizes of tensors must match except in dimension 1. Expected size 256 but got size 128 for tensor number 5 in the list.').
Traceback (most recent call last):
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/optuna/study/_optimize.py", line 197, in _run_trial
    value_or_values = func(trial)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 240, in <lambda>
    lambda trial: self.evaluate_architecture(trial, config, train_loader, val_loader),
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 417, in evaluate_architecture
    outputs = model(inputs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 358, in forward
    concatenated = torch.cat(tensors_to_concat, dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 256 but got size 128 for tensor number 5 in the list.
[W 2024-10-08 13:22:27,986] Trial 0 failed with value None.
Error executing job with overrides: []
Traceback (most recent call last):
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 925, in <module>
    main()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 921, in main
    pipeline.train()
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 639, in train
    self.model = self.build_model(config, train_loader, valid_loader)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 339, in build_model
    best_trial = self.nas.optimize(
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 239, in optimize
    study.optimize(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/optuna/study/study.py", line 475, in optimize
    _optimize(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/optuna/study/_optimize.py", line 63, in _optimize
    _optimize_sequential(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/optuna/study/_optimize.py", line 160, in _optimize_sequential
    frozen_trial = _run_trial(study, func, catch)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/optuna/study/_optimize.py", line 248, in _run_trial
    raise func_err
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/optuna/study/_optimize.py", line 197, in _run_trial
    value_or_values = func(trial)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 240, in <lambda>
    lambda trial: self.evaluate_architecture(trial, config, train_loader, val_loader),
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 417, in evaluate_architecture
    outputs = model(inputs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 358, in forward
    concatenated = torch.cat(tensors_to_concat, dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 256 but got size 128 for tensor number 5 in the list.
(sam3) samerkharboush@Mac SAR-MPS-OptunaNasFbnYoloTransformers %