s % HYDRA_FULL_ERROR=1 python train_multilabel.py +config=configs/multilabel_circle_net.yaml
DEBUG:hydra.core.utils:Setting JobRuntime:name=UNKNOWN_NAME
DEBUG:hydra.core.utils:Setting JobRuntime:name=train_multilabel
Dataset valid_sar_cv
        ignore_low_confidence_detections False
        ignore_low_confidence_labels     False
        filter_low_confidence_objects    False
Train dataset
        Unknown vessels 90
        Unknown fishing 11242
Valid dataset
        Unknown vessels 20
        Unknown fishing 2778
Preparing training dataset: 100%|██| 35/35 [00:00<00:00, 161.98it/s]
[DEBUG] self.box_coder in dataset.__init__: MultilabelCircleNetCoder(image_size=(512, 512), stride=4, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
Preparing validation dataset: 100%|██| 7/7 [00:00<00:00, 569.29it/s]
Train instances 13933 dataset 35 sampler 8192
Valid instances 3655 dataset 1039
[DEBUG] After adjustment, encoder.conv_stem.in_channels: 2
head_output_channels: 6
yolo_output_channels: 21
classifier_output_channels: 2
total_input_channels: 29
head_output_channels: 6
yolo_output_channels: 21
classifier_output_channels: 2
total_input_channels: 29
[DEBUG] model.box_coder after initialization: MultilabelCircleNetCoder(image_size=[768, 768], stride=1, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
BoxCoder
MultilabelCircleNetCoder(image_size=[768, 768], stride=1, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
...

DEBUG:rasterio._io:Window: Window(col_off=4096, row_off=2048, width=2048, height=2048)
DEBUG:rasterio._io:IO window xoff=4096.0 yoff=2048.0 width=2048.0 height=2048.0
[DEBUG] Batch 0 keys: dict_keys(['INPUT_INDEX_KEY', 'INPUT_IMAGE_KEY', 'INPUT_FOLDER_KEY', 'TARGET_SAMPLE_WEIGHT', 'CENTERNET_TARGET_OBJECTNESS_MAP', 'CENTERNET_TARGET_VESSEL_MAP', 'CENTERNET_TARGET_FISHING_MAP', 'CENTERNET_TARGET_SIZE', 'CENTERNET_TARGET_OFFSET', 'TARGET_CENTERS_KEY', 'TARGET_FISHING_KEY', 'TARGET_VESSEL_KEY', 'TARGET_LENGTHS_KEY', 'INPUT_SCENE_ID_KEY', 'INPUT_SCENE_CROP_KEY', 'INPUT_CONFIDENCE_KEY', 'INPUT_LOCATION_ID_KEY', 'INPUT_IS_NEAR_SHORE_KEY'])
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x345e519a0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x345e51640> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x345e51640>.
...

DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x345e51e80>
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
(1336.0, 1495.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(1240.0, 1508.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
Error executing job with overrides: ['+config=configs/multilabel_circle_net.yaml']
Traceback (most recent call last):
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 640, in <module>
    main()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 636, in main
    CenternetPipeline(cfg).train()
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 481, in train
    model = self.build_model(config, self.loaders)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 221, in build_model
    model = nas.search(config.get('nas', {}).get('num_iterations', 4), loaders['train'], loaders['valid'], self.device)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 38, in search
    scores = [self.evaluation_function(model, train_loader, val_loader, device) for model in population]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 38, in <listcomp>
    scores = [self.evaluation_function(model, train_loader, val_loader, device) for model in population]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 589, in evaluate_architecture
    gt_boxes = batch['boxes'].to(self.device)
KeyError: 'boxes'
(sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %