the log is so long after applying your suggestions and modifications but here's the most important snippet of the log

python train_multilabel.py +config=configs/multilabel_circle_net.yaml
DEBUG:hydra.core.utils:Setting JobRuntime:name=UNKNOWN_NAME
DEBUG:hydra.core.utils:Setting JobRuntime:name=train_multilabel
Dataset valid_sar_cv
        ignore_low_confidence_detections False
        ignore_low_confidence_labels     False
        filter_low_confidence_objects    False
Train dataset
        Unknown vessels 90
        Unknown fishing 11242
Valid dataset
        Unknown vessels 20
        Unknown fishing 2778
Preparing training dataset: 100%|█████████████████████| 35/35 [00:00<00:00, 160.98it/s]
[DEBUG] self.box_coder in dataset.__init__: MultilabelCircleNetCoder(image_size=(512, 512), stride=4, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
Preparing validation dataset: 100%|█████████████████████| 7/7 [00:00<00:00, 583.49it/s]
Train instances 13933 dataset 35 sampler 8192
Valid instances 3655 dataset 1039
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 528, in <module>
    main()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 119, in run
    ret = run_job(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 524, in main
    CenternetPipeline(cfg).train()
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 415, in train
    model = self.build_model(config, self.loaders)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 202, in build_model
    model = super().build_model(config)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/pipeline.py", line 395, in build_model
    model = self.get_model(config["model"])
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 379, in get_model
    model: nn.Module = get_detection_model(model_config)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/factory.py", line 27, in get_detection_model
    return instantiate(model_config, _recursive_=False)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
    return _target_(*args, **kwargs)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 198, in from_config
    encoder_module = CustomTimmEncoder(
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/custom_encoder.py", line 10, in __init__
    self.encoder = timm.create_model(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/_factory.py", line 117, in create_model
    model = create_fn(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/_registry.py", line 143, in _fn
    warnings.warn(f'Mapping deprecated model name {deprecated_name} to current {current_name}.', stacklevel=2)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/warnings.py", line 109, in _showwarnmsg
    sw(msg.message, msg.category, msg.filename, msg.lineno,
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 53, in warn_with_traceback
    traceback.print_stack(file=log)
/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/_factory.py:117: UserWarning:

Mapping deprecated model name tf_efficientnet_b5_ns to current tf_efficientnet_b5.ns_jft_in1k.

[DEBUG] After adjustment, encoder.conv_stem.in_channels: 2
[DEBUG] model.box_coder after initialization: MultilabelCircleNetCoder(image_size=[768, 768], stride=1, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
BoxCoder
MultilabelCircleNetCoder(image_size=[768, 768], stride=1, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x332781100>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x32b9a9190>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x33df711c0>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x33ae691f0>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
...
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x328a24b20>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=22622, row_off=20388, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=22622.0 yoff=20388.0 width=512.0 height=512.0
Decoded features length: 4
[DEBUG] decoded_features[0] shape: torch.Size([8, 512, 32, 32])
[DEBUG] decoded_features[1] shape: torch.Size([8, 384, 64, 64])
[DEBUG] decoded_features[2] shape: torch.Size([8, 256, 128, 128])
[DEBUG] decoded_features[3] shape: torch.Size([8, 128, 256, 256])
[DEBUG] feature_maps in FPNFuse: type=<class 'list'>
[DEBUG] Valid feature_maps[0] shape: torch.Size([8, 512, 32, 32])
[DEBUG] Valid feature_maps[1] shape: torch.Size([8, 384, 64, 64])
[DEBUG] Valid feature_maps[2] shape: torch.Size([8, 256, 128, 128])
[DEBUG] Valid feature_maps[3] shape: torch.Size([8, 128, 256, 256])
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x33f978c40>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x33f978df0> options
...
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x33f978e80>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=7086, row_off=13298, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=7086.0 yoff=13298.0 width=512.0 height=512.0
[DEBUG] fpn_output shape: torch.Size([8, 1280, 32, 32])
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x329eec130>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x32f69bcd0> options
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x33f978a30>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=14347, row_off=16538, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=14347.0 yoff=16538.0 width=512.0 height=512.0
[DEBUG] Decoder stage 1
Using encoder feature at index 2
x.shape after concat at stage 1: torch.Size([8, 576, 64, 64])
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x33f978a30>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x33f978bb0> options
...
DEBUG:rasterio._io:IO window xoff=19808.0 yoff=13617.0 width=512.0 height=512.0
Using encoder feature at index 1
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x32e2fe9d0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x32f69bac0> options
...
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x32f69bc10>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x32f69bca0>
DEBUG:rasterio._base:Sharing flag: 0
encoder_feature.shape after resizing at stage 3: torch.Size([8, 24, 256, 256])
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x31a5238e0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x31a523e20> options

...
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x31a523f70>
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=19808, row_off=13617, width=512, height=512)
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:IO window xoff=19808.0 yoff=13617.0 width=512.0 height=512.0
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=7560, row_off=9880, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=7560.0 yoff=9880.0 width=512.0 height=512.0
x.shape after concat at stage 3: torch.Size([8, 280, 256, 256])
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x328a24b20>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x328a5c400> options
DEBUG:rasterio._env:
...
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x32f69db80>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=25690, row_off=563, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=25690.0 yoff=563.0 width=512.0 height=512.0
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x32f69db80>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x329eec040> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x329eec040>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x32f69db80>
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
(255.0, 245.0, 1.6740000247955322) [<class 'float'>, <class 'float'>, <class 'float'>]
Error executing job with overrides: ['+config=configs/multilabel_circle_net.yaml']
Traceback (most recent call last):
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 524, in main
    CenternetPipeline(cfg).train()
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 415, in train
    model = self.build_model(config, self.loaders)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 206, in build_model
    model = nas.search(config.nas.num_iterations, loaders['train'], loaders['valid'])
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 36, in search
    scores = [self.evaluate_model(model, train_loader, val_loader) for model in population]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 36, in <listcomp>
    scores = [self.evaluate_model(model, train_loader, val_loader) for model in population]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 112, in evaluate_model
    outputs = model(inputs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 169, in forward
    neck_features = self.neck(decoded_features)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/necks.py", line 49, in forward
    return self.out_conv(fpn_output)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 458, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 454, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [256, 1024, 1, 1], expected input[8, 1280, 32, 32] to have 1024 channels, but got 1280 channels instead

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
(sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %