DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33239cd00>
[DEBUG] Channel 'vh' image shape before transformations: (512, 512)
[DEBUG] Channel 'vh' image shape after individual_transforms: (512, 512)
[DEBUG] Channel 'vh' image shape after normalization: (512, 512)
[DEBUG] Channel 'vv' image shape before transformations: (512, 512)
[DEBUG] Channel 'vv' image shape after individual_transforms: (512, 512)
[DEBUG] Channel 'vv' image shape after normalization: (512, 512)
[DEBUG] Number of channels: 2
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
[DEBUG] Combined image shape after np.dstack: (512, 512, 2)
[DEBUG] Keypoints being passed to albumentations:
(237.0, 250.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(231.0, 226.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(238.0, 223.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(245.0, 217.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(267.0, 255.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(260.0, 260.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(261.0, 248.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(252.0, 209.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(260.0, 204.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(289.0, 237.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(281.0, 243.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(274.0, 249.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x33e6e3d60>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x33e6e3dc0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x33e6e3dc0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33e6e3d60>
[DEBUG] Channel 'vh' image shape before transformations: (512, 512)
[DEBUG] Channel 'vh' image shape after individual_transforms: (512, 512)
[DEBUG] Channel 'vh' image shape after normalization: (512, 512)
[DEBUG] Channel 'vv' image shape before transformations: (512, 512)
[DEBUG] Channel 'vv' image shape after individual_transforms: (512, 512)
[DEBUG] Channel 'vv' image shape after normalization: (512, 512)
[DEBUG] Number of channels: 2
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
[DEBUG] Combined image shape after np.dstack: (512, 512, 2)
[DEBUG] Keypoints being passed to albumentations:
(36.0, 37.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(25.0, 69.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(76.0, 120.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(37.0, 121.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(2.0, 172.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(259.0, 264.0, 2.214465379714966) [<class 'float'>, <class 'float'>, <class 'float'>]
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfee0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfee0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfe80> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfe80>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000
DEBUG:rasterio._base:Dataset <open DatasetReader name='/Volumes/Crucial/xview3_dataset/full_dataset/full/validation/5e9a2c1bcf179e9bv/VH_dB.tif' mode='r'> is started.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfe80> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfe80>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfe80> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfe80>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=17311, row_off=22200, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=17311.0 yoff=22200.0 width=512.0 height=512.0
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfe80> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfe80>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfee0> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfee0>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000
DEBUG:rasterio._base:Dataset <open DatasetReader name='/Volumes/Crucial/xview3_dataset/full_dataset/full/validation/5e9a2c1bcf179e9bv/VV_dB.tif' mode='r'> is started.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfee0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfee0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dcfe50>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfee0> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfee0>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=17311, row_off=22200, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=17311.0 yoff=22200.0 width=512.0 height=512.0
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dc2a60>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfee0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfee0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dc2a60>
[DEBUG] Channel 'vh' image shape before transformations: (512, 512)
[DEBUG] Channel 'vh' image shape after individual_transforms: (512, 512)
[DEBUG] Channel 'vh' image shape after normalization: (512, 512)
[DEBUG] Channel 'vv' image shape before transformations: (512, 512)
[DEBUG] Channel 'vv' image shape after individual_transforms: (512, 512)
[DEBUG] Channel 'vv' image shape after normalization: (512, 512)
[DEBUG] Number of channels: 2
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
[DEBUG] Combined image shape after np.dstack: (512, 512, 2)
[DEBUG] Keypoints being passed to albumentations:
(447.0, 95.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(437.0, 95.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(463.0, 193.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(453.0, 193.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(327.0, 15.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(316.0, 16.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(348.0, 106.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(482.0, 291.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(472.0, 290.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(232.0, 37.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(219.0, 31.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(281.0, 321.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(267.0, 228.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(268.0, 319.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(257.0, 230.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(503.0, 383.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(493.0, 389.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(301.0, 422.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(292.0, 428.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(366.0, 213.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(250.0, 126.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(238.0, 128.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(384.0, 303.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(375.0, 311.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(420.0, 508.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(400.0, 406.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(391.0, 409.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
Error executing job with overrides: ['+config=configs/multilabel_circle_net.yaml']
Traceback (most recent call last):
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 529, in <module>
    main()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 525, in main
    CenternetPipeline(cfg).train()
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 416, in train
    model = self.build_model(config, self.loaders)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 207, in build_model
    model = nas.search(config.nas.num_iterations, loaders['train'], loaders['valid'])
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 35, in search
    scores = [self.evaluate_model(model, train_loader, val_loader) for model in population]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 35, in <listcomp>
    scores = [self.evaluate_model(model, train_loader, val_loader) for model in population]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 114, in evaluate_model
    outputs = model(inputs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 134, in forward
    neck_features = self.neck(decoded_features)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/necks.py", line 48, in forward
    return self.fpn(feature_maps)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/pytorch_toolbelt/modules/fpn.py", line 126, in forward
    layers.append(F.interpolate(f, size=dst_size, mode=self.mode, align_corners=self.align_corners))
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/functional.py", line 4110, in interpolate
    raise NotImplementedError(
NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 4D) for the modes: nearest | linear | bilinear | bicubic | trilinear | area | nearest-exact (got [512, 384, 256, 128])
(sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %