(sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers % HYDRA_FULL_ERROR=1 python train_multilabel3.py
DEBUG:hydra.core.utils:Setting JobRuntime:name=UNKNOWN_NAME
DEBUG:hydra.core.utils:Setting JobRuntime:name=train_multilabel3
Dataset full_4fold
        ignore_low_confidence_detections True
        ignore_low_confidence_labels     True
        filter_low_confidence_objects    False
Train dataset
        Unknown vessels 7788
        Unknown fishing 31133
Valid dataset
        Unknown vessels 36
        Unknown fishing 3934
Preparing training dataset: 100%|█████████████████████████████████████████████████████████████████████████████████████| 425/425 [00:02<00:00, 188.56it/s]
Preparing validation dataset: 100%|█████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 594.68it/s]
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x3461637c0>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x34720f760>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x33cc0a760>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x34720f730> created
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x346163790> created
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x34446b7c0>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x33cc0a730> created
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x34446b790> created
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:GDAL_DATA found in environment.
....

DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=21276, row_off=5900, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=21276.0 yoff=5900.0 width=512.0 height=512.0
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x34720f730>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x34720f6d0> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x34720f6d0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x34720f730>
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
(297.0, 256.0, 2.499000072479248) [<class 'float'>, <class 'float'>, <class 'float'>]
Available keys in model config: dict_keys(['_target_', 'slug', 'weights', 'pretrained', 'output_stride', 'image_size', 'decoder', 'encoder', 'nas', 'num_channels', 'num_classes', 'anchors', 'input_upsample_factor', 'neck_out_channels', 'box_coder', 'head', 'config'])
Complete model configuration:
 _target_: xview3.centernet.models.CenterNetUNetModel.from_config
slug: b5_unet_s2
weights: IMAGENET1K_V1
pretrained: true
output_stride: 32
image_size:
- 1024
- 1024
decoder:
  _target_: xview3.centernet.models.custom_encoder.DebugUNetDecoder
  encoder_channels:
  - 32
  - 24
  - 40
  - 112
  - 320
  channels:
  - 512
  - 384
  - 256
  - 128
  - 64
  unet_block:
    _target_: xview3.centernet.models.unet_blocks.UnetBlock
    _partial_: true
  upsample_block:
    _target_: torch.nn.Upsample
    scale_factor: 2
    mode: bilinear
encoder:
  _target_: pytorch_toolbelt.modules.encoders.TimmEfficientNetV2
  model_name: efficientnetv2_rw_s
  pretrained: true
  activation: silu
  drop_rate: 0.1
  drop_path_rate: 0.1
  layers:
  - 1
  - 2
  - 3
  - 4
nas:
  enabled: true
  search_space:
    num_layers:
    - 2
    - 5
    num_filters:
    - 64
    - 128
    - 256
    kernel_size:
    - 3
    - 5
    activation:
    - relu
    - leaky_relu
    dropout_rate_max: 0.5
  num_trials: 50
num_channels: 2
num_classes: 2
anchors:
- - 10
  - 13
- - 16
  - 30
- - 33
  - 23
input_upsample_factor: 1
neck_out_channels: 256
box_coder:
  _target_: xview3.centernet.MultilabelCircleNetCoder
  image_size:
  - 1024
  - 1024
  max_objects: 512
  heatmap_encoding: umich
  labels_encoding: circle
  ignore_value:
    _target_: xview3.ignore_value
  fixed_radius: 3
  labels_radius: 1
  ignore_low_confidence_detections: ${dataset.ignore_low_confidence_detections}
head:
  decoupled_head:
    _target_: xview3.centernet.models.heads.DecoupledHeadGroupNormLateShuffle
    channels: 256
    classifier_dim: 128
    objectness_dim: 128
    size_dim: 128
    offset_dim: 16
    dropout_rate: 0.1
    activation: silu
    num_blocks: 3
  yolo_head:
    _target_: xview3.centernet.models.heads.yolo_head.YOLOHead
    num_classes: 2
    anchors:
    - - 10
      - 13
    - - 16
      - 30
    - - 33
      - 23
    input_image_size:
    - 640
    - 640
    stride: 32
config:
  slug: b5_unet_s2
  activation: silu
  num_channels: ${dataset.num_channels}
  num_extra_blocks: 0
  encoder:
    _target_: pytorch_toolbelt.modules.encoders.TimmB5Encoder
    pretrained: true
    activation: silu
    layers:
    - 1
    - 2
    - 3
    - 4
  decoder:
    upsample_type: shuffle
    block_type: IRBlock
    channels:
    - 256
    - 384
    - 512
  head:
    _target_: xview3.centernet.models.heads.DecoupledHeadGroupNormLateShuffle
    classifier_dim: 128
    objectness_dim: 128
    size_dim: 64
    offset_dim: 16
    dropout_rate: 0.1
    activation: silu
    num_groups: 4
    num_blocks: 3
  box_coder:
    _target_: xview3.centernet.MultilabelCircleNetCoder
    image_size:
    - 1024
    - 1024
    max_objects: 512
    heatmap_encoding: umich
    labels_encoding: circle
    ignore_value:
      _target_: xview3.ignore_value
    fixed_radius: 3
    labels_radius: 1
    ignore_low_confidence_detections: ${dataset.ignore_low_confidence_detections}

[DEBUG] Number of heads instantiated: 2
[DEBUG] Head 0 type: <class 'xview3.centernet.models.heads.decoupled_head.DecoupledHeadGroupNormLateShuffle'>
[DEBUG] Head 1 type: <class 'xview3.centernet.models.heads.yolo_head.YOLOHead'>
[2024-10-05 00:10:28,450][timm.models._builder][INFO] - Loading pretrained weights from Hugging Face hub (timm/efficientnetv2_rw_s.ra2_in1k)
[2024-10-05 00:10:28,992][timm.models._hub][INFO] - [timm/efficientnetv2_rw_s.ra2_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
[2024-10-05 00:10:29,097][timm.models._builder][WARNING] - Unexpected keys (bn2.bias, bn2.num_batches_tracked, bn2.running_mean, bn2.running_var, bn2.weight, classifier.bias, classifier.weight, conv_head.weight) found while loading pretrained weights. This may be expected if model is being adapted.
[2024-10-05 00:10:29,346][timm.models._builder][INFO] - Loading pretrained weights from Hugging Face hub (timm/efficientnetv2_rw_s.ra2_in1k)
[2024-10-05 00:10:29,501][timm.models._hub][INFO] - [timm/efficientnetv2_rw_s.ra2_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
[2024-10-05 00:10:29,515][timm.models._builder][INFO] - Converted input conv conv_stem pretrained weights from 3 to 2 channel(s)
[2024-10-05 00:10:29,530][timm.models._builder][WARNING] - Unexpected keys (bn2.bias, bn2.num_batches_tracked, bn2.running_mean, bn2.running_var, bn2.weight, classifier.bias, classifier.weight, conv_head.weight) found while loading pretrained weights. This may be expected if model is being adapted.
[2024-10-05 00:10:29,531][xview3.centernet.models.centernet_unet][INFO] - Adjusted encoder's first conv layer 'initial_conv' to have 2 input channels.
[DEBUG] Initializing SwinTransformerEncoder with window_size=4, img_size=16
[DEBUG] SwinTransformerEncoder initialized with img_size=16 and window_size=4
[DEBUG] CenterNetUNetModel initialized with window_size=4 and img_size=16
[2024-10-05 00:10:34,380][__main__][INFO] - Total parameters in the model: {'total': 264872087, 'trainable': 264872087, 'encoder': 21657072, 'decoder': 14193536}
1/100 * Epoch (train):   0% 0/27 [00:00<?, ?it/s]DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x33bc137f0>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x33ca93820>
....

DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x3692d4550> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x3692d4550>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x3692d4d90>
DEBUG:rasterio._io:Output nodata value read from file: -32768.0
DEBUG:rasterio._io:Output nodata values: [-32768.0]
DEBUG:rasterio._io:all_valid: False
DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],)
DEBUG:rasterio._io:Jump straight to _read()
DEBUG:rasterio._io:Window: Window(col_off=23797, row_off=3633, width=512, height=512)
DEBUG:rasterio._io:IO window xoff=23797.0 yoff=3633.0 width=512.0 height=512.0
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x3692d4d90>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x3692d4550> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x3692d4550>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x3692d4d90>
[DEBUG] Channel 0 shape: (512, 512)
[DEBUG] Channel 1 shape: (512, 512)
(274.0, 274.0, 1.5899999141693115) [<class 'float'>, <class 'float'>, <class 'float'>]
[DEBUG] Feature 0 shape: torch.Size([16, 24, 256, 256])
[DEBUG] Feature 1 shape: torch.Size([16, 48, 128, 128])
[DEBUG] Feature 2 shape: torch.Size([16, 64, 64, 64])
[DEBUG] Feature 3 shape: torch.Size([16, 160, 32, 32])
[DEBUG] Feature 4 shape: torch.Size([16, 272, 16, 16])
[DEBUG] SwinTransformer.forward_features: window_size=4, input_shape=torch.Size([16, 272, 16, 16])
Error executing job with overrides: []
Traceback (most recent call last):
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 806, in <module>
    main()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 802, in main
    pipeline.train()
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel3.py", line 621, in train
    runner.train(
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/runners/runner.py", line 170, in train
    distributed_cmd_run(self.run_experiment, distributed)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/utils/scripts.py", line 176, in distributed_cmd_run
    worker_fn(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 929, in run_experiment
    self._run_event("on_exception")
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 724, in _run_event
    getattr(callback, event)(self)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/callbacks/exception.py", line 34, in on_exception
    raise exception
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 917, in run_experiment
    self._run_stage(stage)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 885, in _run_stage
    self._run_epoch(stage=stage, epoch=self.epoch)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 864, in _run_epoch
    self._run_loader(loader)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 801, in _run_loader
    self._run_batch(batch)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/core/runner.py", line 775, in _run_batch
    self._handle_batch(batch=batch)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/runners/supervised.py", line 144, in _handle_batch
    self.output = self.forward(batch)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/runners/supervised.py", line 131, in forward
    output = self._process_input(batch, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/catalyst/runners/supervised.py", line 94, in _process_input_str
    output = self.model(batch[self.input_key], **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 286, in forward
    transformer_features = self.transformer_encoder(feature_maps[-1])
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/transformer_classifier.py", line 148, in forward
    features = self.swin(x)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 838, in forward
    x = self.forward_features(x)
  File "/Volumes/Crucial/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/transformer_classifier.py", line 27, in forward_features
    return super().forward_features(x)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 830, in forward_features
    x = self.layers(x)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/container.py", line 219, in forward
    input = module(input)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 559, in forward
    x = self.blocks(x)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/container.py", line 219, in forward
    input = module(input)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 406, in forward
    x = x + self.drop_path1(self._attn(self.norm1(x)))
  File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 376, in _attn
    pad_h = (self.window_size[0] - H % self.window_size[0]) % self.window_size[0]
ZeroDivisionError: integer division or modulo by zero
1/100 * Epoch (train):   0% 0/27 [00:23<?, ?it/s]
(sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %