now it's running perfectly but how I know if it's training or stuck! as the log still updating with many lines , the most noticeable are these info

HYDRA_FULL_ERROR=1 python train_multilabel.py +config=configs/multilabel_circle_net.yaml
DEBUG:hydra.core.utils:Setting JobRuntime:name=UNKNOWN_NAME
DEBUG:hydra.core.utils:Setting JobRuntime:name=train_multilabel
Dataset valid_sar_cv
        ignore_low_confidence_detections False
        ignore_low_confidence_labels     False
        filter_low_confidence_objects    False
Train dataset
        Unknown vessels 90
        Unknown fishing 11242
Valid dataset
        Unknown vessels 20
        Unknown fishing 2778
Preparing training dataset: 100%|█| 35/35 [00:00<00:00, 163.72it
[DEBUG] self.box_coder in dataset.__init__: MultilabelCircleNetCoder(image_size=(512, 512), stride=4, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
Preparing validation dataset: 100%|█| 7/7 [00:00<00:00, 549.31it
Train instances 13933 dataset 35 sampler 8192
Valid instances 3655 dataset 1039
Losses
   losses/ MultilabelCenterTrimmedLoss weight 1.0
     target ['CENTERNET_TARGET_OBJECTNESS_MAP', 'CENTERNET_TARGET_VESSEL_MAP', 'CENTERNET_TARGET_FISHING_MAP', 'CENTERNET_TARGET_SIZE', 'CENTERNET_TARGET_OFFSET']
     output ['CENTERNET_OUTPUT_OBJECTNESS_MAP', 'CENTERNET_OUTPUT_VESSEL_MAP', 'CENTERNET_OUTPUT_FISHING_MAP', 'CENTERNET_OUTPUT_SIZE', 'CENTERNET_OUTPUT_OFFSET']
[DEBUG] After adjustment, encoder.conv_stem.in_channels: 2
head_output_channels: 6
yolo_output_channels: 21
classifier_output_channels: 2
total_input_channels: 29
head_output_channels: 6
yolo_output_channels: 21
classifier_output_channels: 2
total_input_channels: 29
[DEBUG] model.box_coder after initialization: MultilabelCircleNetCoder(image_size=[768, 768], stride=1, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
BoxCoder
MultilabelCircleNetCoder(image_size=[768, 768], stride=1, max_objects=512, labels_encoding=circle, heatmap_encoding=umich, fixed_radius=None, ignore_index=-1, ignore_low_confidence_detections=False)
...
asterio._env.GDALEnv object at 0x33dec98b0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33dec9a60>
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
(1336.0, 1495.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(1240.0, 1508.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
Added new layer to encoder: Conv2d(6, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Added new layer to encoder: Conv2d(6, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Added new layer to encoder: Conv2d(6, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Added new layer to encoder: Conv2d(6, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Added new layer to encoder: Conv2d(6, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
--- NAS Iteration 1/4 ---
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.session:Could not import boto3, continuing with reduced functionality.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x32968ea00>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x339b4e9d0>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x34578e9d0>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x331a1aa90>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x339b4ea00> created
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x32968ea30> created
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x34578ea00> created
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x331a1aac0> created
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._filepath:Installing FilePath filesystem handler plugin...
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x339b4ea00>.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x331a1aac0>.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x32968ea30>.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x34578ea00>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x339b4e9d0>
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x331a1aa90>
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x32968ea00>
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x34578e9d0>
DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Sharing flag: 0
DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000
DEBUG:rasterio._base:Dataset <open DatasetReader name='/Volumes/Crucial/xview3_dataset/full_dataset/validation/4a97701b4bd81bf7v/VH_dB.tif' mode='r'> is started.
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x34578e9d0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x34578ea00> options
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x34578ea00>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x34578e9d0>
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x34578ea90>
DEBUG:rasterio.env:Starting outermost env
...
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x325dfa400>
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
[DEBUG] Feature 0 shape: torch.Size([4, 24, 1023, 1023])
[DEBUG] Feature 1 shape: torch.Size([4, 40, 512, 512])
[DEBUG] Feature 2 shape: torch.Size([4, 64, 256, 256])
[DEBUG] Feature 3 shape: torch.Size([4, 176, 128, 128])
[DEBUG] Feature 4 shape: torch.Size([4, 512, 64, 64])
[DEBUG] Number of encoder features: 5
[DEBUG] Decoder stage 0
Using encoder feature at index 3
x.shape after concat at stage 0: torch.Size([4, 688, 128, 128])
[DEBUG] Decoder stage 1
Using encoder feature at index 2
x.shape after concat at stage 1: torch.Size([4, 576, 256, 256])
[DEBUG] Decoder stage 2
Using encoder feature at index 1
x.shape after concat at stage 2: torch.Size([4, 424, 512, 512])
[DEBUG] Decoder stage 3
Using encoder feature at index 0
encoder_feature.shape after resizing at stage 3: torch.Size([4, 24, 1024, 1024])
x.shape after concat at stage 3: torch.Size([4, 280, 1024, 1024])
[DEBUG] feature_maps in FPNFuse: type=<class 'list'>
[DEBUG] Valid feature_maps[0] shape: torch.Size([4, 512, 128, 128])
[DEBUG] Valid feature_maps[1] shape: torch.Size([4, 384, 256, 256])
[DEBUG] Valid feature_maps[2] shape: torch.Size([4, 256, 512, 512])
[DEBUG] Valid feature_maps[3] shape: torch.Size([4, 128, 1024, 1024])
yolo_output shape: torch.Size([4, 21, 128, 128])
objectness_map shape: torch.Size([4, 1, 256, 256])
vessel_map shape: torch.Size([4, 1, 256, 256])
fishing_map shape: torch.Size([4, 1, 256, 256])
size_map shape: torch.Size([4, 1, 256, 256])
offset_map shape: torch.Size([4, 2, 256, 256])
yolo_output shape: torch.Size([4, 21, 128, 128])
classification_output shape: torch.Size([4, 2, 128, 128])
yolo_output upsampled to: torch.Size([4, 21, 256, 256])
classification_output upsampled to: torch.Size([4, 2, 256, 256])
[2024-09-30 10:10:26,595][__main__][ERROR] - Expected keys 'CENTERNET_OUTPUT_YOLO' and 'CENTERNET_TARGET_YOLO' not found in model outputs or encoded targets.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x32dbbc040>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x32dbbc8e0> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
...

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x32dbbc880>
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
[DEBUG] Feature 0 shape: torch.Size([4, 24, 1023, 1023])
[DEBUG] Feature 1 shape: torch.Size([4, 40, 512, 512])
[DEBUG] Feature 2 shape: torch.Size([4, 64, 256, 256])
[DEBUG] Feature 3 shape: torch.Size([4, 176, 128, 128])
[DEBUG] Feature 4 shape: torch.Size([4, 512, 64, 64])
[DEBUG] Number of encoder features: 5
[DEBUG] Decoder stage 0
Using encoder feature at index 3
x.shape after concat at stage 0: torch.Size([4, 688, 128, 128])
[DEBUG] Decoder stage 1
Using encoder feature at index 2
x.shape after concat at stage 1: torch.Size([4, 576, 256, 256])
[DEBUG] Decoder stage 2
Using encoder feature at index 1
x.shape after concat at stage 2: torch.Size([4, 424, 512, 512])
[DEBUG] Decoder stage 3
Using encoder feature at index 0
encoder_feature.shape after resizing at stage 3: torch.Size([4, 24, 1024, 1024])
x.shape after concat at stage 3: torch.Size([4, 280, 1024, 1024])
[DEBUG] feature_maps in FPNFuse: type=<class 'list'>
[DEBUG] Valid feature_maps[0] shape: torch.Size([4, 512, 128, 128])
[DEBUG] Valid feature_maps[1] shape: torch.Size([4, 384, 256, 256])
[DEBUG] Valid feature_maps[2] shape: torch.Size([4, 256, 512, 512])
[DEBUG] Valid feature_maps[3] shape: torch.Size([4, 128, 1024, 1024])
yolo_output shape: torch.Size([4, 21, 128, 128])
objectness_map shape: torch.Size([4, 1, 256, 256])
vessel_map shape: torch.Size([4, 1, 256, 256])
fishing_map shape: torch.Size([4, 1, 256, 256])
size_map shape: torch.Size([4, 1, 256, 256])
offset_map shape: torch.Size([4, 2, 256, 256])
yolo_output shape: torch.Size([4, 21, 128, 128])
classification_output shape: torch.Size([4, 2, 128, 128])
yolo_output upsampled to: torch.Size([4, 21, 256, 256])
classification_output upsampled to: torch.Size([4, 2, 256, 256])
[2024-09-30 10:11:27,075][__main__][ERROR] - Expected keys 'CENTERNET_OUTPUT_YOLO' and 'CENTERNET_TARGET_YOLO' not found in model outputs or encoded targets.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x32ccba280>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x32ccba580> created
DEBUG:rasterio._env:GDAL_DATA found in environment.
DEBUG:rasterio._env:PROJ_DATA found in environment.
DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x32ccba580>.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x32ccba280>
DEBUG:rasterio._base:Sharing flag: 0
....

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x32ccba2e0>
[DEBUG] Channel 0 shape: (2048, 2048)
[DEBUG] Channel 1 shape: (2048, 2048)
(853.0, 1182.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(1282.0, 318.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(1114.0, 1927.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(784.0, 1689.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>]
(846.0, 1683.0, 4.918635368347168) [<class 'float'>, <class 'float'>, <class 'float'>]
(966.0, 1802.0, 7.295952796936035) [<class 'float'>, <class 'float'>, <class 'float'>]
(937.0, 1805.0, 3.836611270904541) [<class 'float'>, <class 'float'>, <class 'float'>]
(1118.0, 1911.0, 6.659096717834473) [<class 'float'>, <class 'float'>, <class 'float'>]
(742.0, 1742.0, 8.108647346496582) [<class 'float'>, <class 'float'>, <class 'float'>]
(969.0, 1556.0, 8.419084548950195) [<class 'float'>, <class 'float'>, <class 'float'>]
No scenes to process for predictions.