to be able to give the correct solution you need to see these files
1-config
2-centernet_unet.py
3-yolo_head.py
4-yolo_decoder.py
5-decoupled_head.py
6-unet_block.py
7-custom_encoder.py
8-necks.py
9-transformer_classifier.py
and before all you should understand that
This project is a complex integrated multi-model system designed for vessel detection and fishing activity classification using SAR (Synthetic Aperture Radar) TIFF images as input. The system leverages several advanced deep learning architectures and techniques to achieve high accuracy and efficiency in maritime surveillance and monitoring.
input csv labels train, validation
train
detect_lat,detect_lon,vessel_length_m,source,detect_scene_row,detect_scene_column,is_vessel,is_fishing,distance_from_shore_km,scene_id,confidence,top,left,bottom,right,detect_id
5.66292355123629,4.84242888997755,,ais,16722,22703,,,9999.99,e42a50089e03990ft,LOW,,,,,e42a50089e03990ft_005.66292355123628965430_004.84242888997755027702
5.83078557395605,4.79439446548728,,ais,14867,22165,False,,9999.99,e42a50089e03990ft,HIGH,,,,,e42a50089e03990ft_005.83078557395605034941_004.79439446548728032127
5.65028885294943,5.07650030801318,,ais,16853,25297,False,,4.1204849040804055,e42a50089e03990ft,HIGH,,,,,e42a50089e03990ft_005.65028885294942995188_005.07650030801317964091
validation
detect_lat,detect_lon,vessel_length_m,source,detect_scene_row,detect_scene_column,is_vessel,is_fishing,distance_from_shore_km,scene_id,confidence,top,left,bottom,right,detect_id
44.28796374,13.04173288,,manual,9923,23071,True,,9999.99,264ed833a13b7f2av,LOW,9918.0,23068.0,9928.0,23074.0,264ed833a13b7f2av_044.28796374000000213300_013.04173287999999963915
44.61788761,12.35710138,,manual,6105,17727,True,,6.25220144977796,264ed833a13b7f2av,LOW,6099.0,17725.0,6111.0,17729.0,264ed833a13b7f2av_044.61788760999999681189_012.35710137999999957970
44.28847473,13.7108852,,manual,10023,28410,True,,9999.99,264ed833a13b7f2av,MEDIUM,10018.0,28408.0,10028.0,28412.0,264ed833a13b7f2av_044.28847472999999723697_013.71088519999999988386
Key Components
CenterNet UNet
Combines CenterNet for object detection and U-Net for image segmentation.
YOLO (You Only Look Once)
A real-time object detection system known for its speed and accuracy.
Neural Architecture Search (NAS)
Optimizes model architecture to enhance performance.
Feature Pyramid Network (FPN)
Enables multi-scale feature extraction, improving detection across various object sizes.
Feature Aggregation Network (FAN)
Combines features from different network levels for enriched representation.
Swin Transformer
A hierarchical vision transformer that captures global context effectively.
Optuna
A hyperparameter optimization framework to fine-tune model settings.
Output Format
The system processes SAR images and outputs predictions in a CSV format with the following columns:
vessel_length_m: Length of the detected vessel in meters.
detect_scene_row: Row coordinate in the detection scene.
detect_scene_column: Column coordinate in the detection scene.
scene_id: Identifier for the scene.
is_vessel: Boolean indicating if a vessel is detected.
is_fishing: Boolean indicating if fishing activity is detected.
Sample Output:
vessel_length_m,detect_scene_row,detect_scene_column,scene_id,is_vessel,is_fishing
33.99261,11448,13456,1,True,True
28.166428,15632,12622,1,True,True
32.700775,19454,4840,1,True,True
37.80931,12862,2608,1,True,True
48.392456,4902,5522,1,True,False
The system architecture encompasses model configuration, data augmentation, training processes, and inference pipelines
so please understand all this and remember and respond with done so I can show you the files to deeply review and remember and respond with only 'done' so we can process with find solution
###config
defaults:
self
dataset: full_4fold
augs: flips_light
sampler: balanced
normalization: default
loss: centernet/rfl_bce_mse
model: centernet/b5_unet_s2
optimizer: adamw
scheduler: cos
train: default
runner: default
ema: disabled
torch: default
slug: b5_unet_s2
activation: silu
num_channels: ${dataset.num_channels}
num_extra_blocks: 0
model:
target: xview3.centernet.models.centernet_unet.EnhancedCenterNetUNet
slug: b5_unet_s2
weights: IMAGENET1K_V1
pretrained: True
output_stride: 16
image_size: [2048, 2048]
window_size:
- 4
- 4
num_layers: [2, 5]
num_filters: [64, 128, 256]
kernel_size: [3, 5]
activation: silu
dropout_rate: [0.0, 0.5]
dropout_rate_max: 0.5
optimizer: adamw
learning_rate: 0.001
transformer:
enabled: true
embed_dim: 96
depths: [2, 2, 6, 2]
num_heads: [3, 6, 12, 24]
window_size:
- 4
- 4
mlp_ratio: 4.0
qkv_bias: true
qk_scale: null
drop_rate: 0.0
attn_drop_rate: 0.0
drop_path_rate: 0.1
norm_layer: torch.nn.LayerNorm
use_dropout: false
use_custom_pos_embed: false
decoder:
target: xview3.centernet.models.custom_encoder.DebugUnetDecoder
encoder_channels: [24, 48, 64, 160, 272] #[32, 24, 40, 112, 320]
decoder_channels: [512, 384, 256, 128, 64]
unet_block:
target: xview3.centernet.models.unet_blocks.UnetBlock
partial: true
upsample_block: xview3.centernet.models.custom_encoder.UpsampleBlock
#scale_factor: 2
#mode: bilinear
center: False # Changed from 'bilinear' to 'nearest'
activation: silu
encoder:
target: pytorch_toolbelt.modules.encoders.TimmEfficientNetV2
model_name: efficientnetv2_rw_s
pretrained: True #features_only: True
activation: silu
drop_rate: 0.1
drop_path_rate: 0.1
layers:
- 1 # 4
- 2 # 8
- 3 # 16
- 4 # 32
nas:
enabled: true
search_space:
num_layers: [2, 5]
num_filters: [64, 128, 256]
kernel_size: [3, 5]
activation: [relu, leaky_relu, silu]
dropout_rate: [0.0, 0.5]
dropout_rate_max: 0.5
num_trials: 50
metric: "val_acc"
direction: "maximize"
num_channels: ${dataset.num_channels}
num_classes: 2
anchors:
- [10, 13]
- [16, 30]
- [33, 23]
- [30, 61]
- [62, 45]
- [59, 119]
input_upsample_factor: 1
neck_out_channels: 256
box_coder:
target: xview3.centernet.bboxer.multilabel_circle_coder.MultilabelCircleNetCoder
image_size:
- 2048
- 2048
max_objects: 512
heatmap_encoding: umich
labels_encoding: circle
ignore_value:
target: xview3.ignore_value
fixed_radius: 3
labels_radius: 1
ignore_low_confidence_detections: ${dataset.ignore_low_confidence_detections}
head:
decoupled_head:
target: xview3.centernet.models.heads.DecoupledHeadGroupNormLateShuffle
channels: 256
num_classes: 2
classifier_dim: 128
objectness_dim: 128
size_dim: 128
offset_dim: 16
dropout_rate: 0.1
activation: silu
num_blocks: 3
anchors: [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45], [59, 119]]
yolo_head:
target: xview3.centernet.models.heads.yolo_head.YOLOHead
num_classes: 2
input_channels: 256
anchors: [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45], [59, 119]]
input_image_size: [2048, 2048]
stride: 16
num_anchors_per_scale: 3
fan:
target: xview3.centernet.models.necks.FeatureAggregationNetwork
input_channels: [272] #[32, 48, 64, 128] #in_channels: 272 #out_channels: 256 #num_layers: 2
out_channels: 256
num_layers: 2
upsample_mode: bilinear
use_attention: True
fusion_method: weighted_sum
expected_fan_input_channels: 256
Per-Channel Normalization
normalization:
channels:
vv:
target: xview3.SigmoidNormalization
midpoint: -20.0
temperature: 0.18
vh:
target: xview3.SigmoidNormalization
midpoint: -20.0
temperature: 0.18
bathymetry:
target: xview3.CubicRootNormalization
wind_direction:
target: albumentations.NoOp
wind_speed:
target: albumentations.NoOp
mask:
target: albumentations.NoOp
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
Dataset Configuration
dataset:
normalization:
channels:
vv:
target: xview3.SigmoidNormalization
midpoint: -20.0
temperature: 0.18
vh:
target: xview3.SigmoidNormalization
midpoint: -20.0
temperature: 0.18
bathymetry:
target: xview3.CubicRootNormalization
wind_direction:
target: albumentations.NoOp
wind_speed:
target: albumentations.NoOp
mask:
target: albumentations.NoOp
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
slug: full_4fold
splitter: full
fold: 0
num_folds: 4
Added the following parameters to address references in box_coder
ignore_low_confidence_detections: true
ignore_low_confidence_labels: false
filter_low_confidence_objects: False
num_channels: 2
input_channels:
- vh
- vv
DDP Stuff
local_rank: ${oc.env:LOCAL_RANK, 0}
world_size: ${oc.env:WORLD_SIZE, 0}
experiment: null
seed: 42
deterministic: False
detect_anomaly: False
find_unused: False
transfer: null
checkpoint: null
inference:
channels_last: False
use_traced_model: True
tile_size: 2048
tile_step: 1536
accumulate_on_gpu: False
batch_size: 1
fp16: True
Torch Configuration
torch:
amp:
enabled: false
cudnn:
allow_tf32: True
cuda_allow_tf32: True
detect_anomaly: False
deterministic: False
benchmark: True
Inference and Training Enhancements
activation_after: "after_model"
tta: "d4"
with_offset: True
yolo_decoders: []
Validation Configuration
valid:
image_size: [2048, 2048]
batch_size: 2
num_workers: 2
Training Configuration
train:
slug: default_training
epochs: 4
output_stride: 16
early_stopping:
enabled: true
patience: 2
mode: "max" # Options: 'min' for loss-based, 'max' for accuracy-based
metric: "loss" # The metric to monitor for early stopping
image_size: [2048, 2048]
batch_size: 2
num_workers: 2
num_samples: 8192
persistent_workers: False
model2:
target: xview3.centernet.models.centernet_unet.EnhancedCenterNetUNet
slug: b5_unet_s2
weights: IMAGENET1K_V1
pretrained: True
output_stride: 16
image_size: [2048, 2048]
window_size:
- 4
- 4
num_layers: ${model2.num_layers} # Ensure these are set up for overrides
num_filters: ${model2.num_filters}
kernel_size: ${model2.kernel_size}
activation: ${model2.activation}
dropout_rate: ${model2.dropout_rate}
dropout_rate_max: 0.5
optimizer: ${model2.optimizer}
learning_rate: ${model2.learning_rate}
transformer:
enabled: true
embed_dim: 96
depths: [2, 2, 6, 2]
num_heads: [3, 6, 12, 24]
window_size:
- 4
- 4
mlp_ratio: 4.0
qkv_bias: true
qk_scale: null
drop_rate: 0.0
attn_drop_rate: 0.0
drop_path_rate: 0.1
norm_layer: torch.nn.LayerNorm
use_dropout: false
use_custom_pos_embed: false
decoder:
target: xview3.centernet.models.custom_encoder.DebugUNetDecoder
encoder_channels: [24, 48, 64, 160, 272]
channels: [512, 384, 256, 128, 64]
unet_block:
target: xview3.centernet.models.unet_blocks.UnetBlock
partial: true
upsample_block:
target: torch.nn.Upsample
scale_factor: 2
mode: bilinear
encoder:
target: pytorch_toolbelt.modules.encoders.TimmEfficientNetV2
model_name: efficientnetv2_rw_s
pretrained: True
activation: silu
drop_rate: 0.1
drop_path_rate: 0.1
layers:
- 1 # 4
- 2 # 8
- 3 # 16
- 4 # 32
nas:
enabled: true
search_space:
num_layers: [2, 5]
num_filters: [64, 128, 256]
kernel_size: [3, 5]
activation: [relu, leaky_relu, silu]
dropout_rate: [0.0, 0.5]
dropout_rate_max: 0.5
num_trials: 50
metric: "val_acc"
direction: "maximize"
num_channels: ${dataset.num_channels}
num_classes: 2
anchors: [[10, 13], [16, 30], [33, 23]]
input_upsample_factor: 1
neck_out_channels: 256
box_coder:
target: xview3.centernet.bboxer.multilabel_circle_coder.MultilabelCircleNetCoder
image_size:
- 2048
- 2048
max_objects: 512
heatmap_encoding: umich
labels_encoding: circle
ignore_value:
target: xview3.ignore_value
fixed_radius: 3
labels_radius: 1
ignore_low_confidence_detections: ${dataset.ignore_low_confidence_detections}
head:
decoupled_head:
target: xview3.centernet.models.heads.DecoupledHeadGroupNormLateShuffle
channels: 256
classifier_dim: 128
objectness_dim: 128
size_dim: 128
offset_dim: 16
dropout_rate: 0.1
activation: silu
num_blocks: 3
yolo_head:
target: xview3.centernet.models.heads.yolo_head.YOLOHead
num_classes: 2
anchors: [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45], [59, 119]]
input_image_size: [2048, 2048]
stride: 16
num_anchors_per_scale: 3
fan:
target: xview3.centernet.models.necks.FeatureAggregationNetwork
input_channels: [272] #[32, 48, 64, 128] #in_channels: 272 #out_channels: 256 #num_layers: 2
out_channels: 256
num_layers: 2
upsample_mode: bilinear
use_attention: True
fusion_method: weighted_sum