DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000
DEBUG:rasterio._base:Dataset <open DatasetReader name='/Volumes/Crucial/xview3_dataset/full_dataset/full/validation/9ea5029406691ce4v/VH_dB.tif' mode='r'> is started.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x345c311c0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x345c31a30> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x345c31a30>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x345c311c0>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x341dd0cd0>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x345c31a30> created
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x33c36fca0>
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x33c36fc10> options
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x33c36fc10>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33c36fca0>
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x345c31a30>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x341dd0cd0>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
Traceback (most recent call last):
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 683, in <module>
    main()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 679, in main
    CenternetPipeline(cfg).train()
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 481, in train
    model = self.build_model(config, self.loaders)
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 211, in build_model
    logger.debug(f"Train loader sample: {next(iter(loaders['train']))}")
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1344, in _next_data
    return self._process_data(data)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1370, in _process_data
    data.reraise()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/_utils.py", line 706, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/multilabel_dataset.py", line 141, in build_centernet_sample
    confidences = np.array([list(conf) for conf in confidences], dtype=np.float32)
ValueError: could not convert string to float: 'M'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/dataset/random_crop_dataset.py", line 191, in __getitem__
    return self.build_sample(
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/multilabel_dataset.py", line 361, in build_sample
    return self.build_centernet_sample(
  File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/multilabel_dataset.py", line 143, in build_centernet_sample
    raise ValueError("Confidences array contains non-numeric or inconsistent data.") from e
ValueError: Confidences array contains non-numeric or inconsistent data.

DEBUG:rasterio._io:all_valid: False
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=19067, row_off=19797, width=512, height=512)
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x34572ed30>
DEBUG:rasterio._io:IO window xoff=19067.0 yoff=19797.0 width=512.0 height=512.0
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x34572eca0> options
DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x34572eca0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x34572ed30>
DEBUG:rasterio._base:Dataset <open DatasetReader name='/Volumes/Crucial/xview3_dataset/full_dataset/full/validation/fc2f57371370c521v/VH_dB.tif' mode='r'> is started.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x33ddc0a00>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x33ddc02b0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x33ddc02b0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33ddc0a00>
(168.0, 45.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(229.0, 264.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(238.0, 285.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(236.0, 271.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x33ddc07f0>
(244.0, 276.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(252.0, 281.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(175.0, 49.0, 1.8786169290542603) [<class 'float'>, <class 'float'>, <class 'float'>]
DEBUG:rasterio.env:Starting outermost env
(183.0, 53.0, 1.8199727535247803) [<class 'float'>, <class 'float'>, <class 'float'>]
(190.0, 54.0, 2.3164262771606445) [<class 'float'>, <class 'float'>, <class 'float'>]
(182.0, 36.0, 1.0649999380111694) [<class 'float'>, <class 'float'>, <class 'float'>]
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x33ddc02b0> created
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x33ddc02b0>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x33ddc07f0>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
(45.0, 333.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(267.0, 241.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=22205, row_off=10815, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=22205.0 yoff=10815.0 width=512.0 height=512.0
[DEBUG] Build Sample - Confidences Type: <class 'numpy.ndarray'>
[DEBUG] Confidences Dtype: object
[DEBUG] Confidences Shape: (2,)
[DEBUG] Build Sample - Confidences Type: <class 'numpy.ndarray'>
[DEBUG] Confidences Dtype: object
[DEBUG] Confidences Shape: (10,)
(sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %