the process would be blocking in Validation step
Problem:
I have a problem. when i train the model, the process would be blocking in validation step(or validation sanity check) but in the training step, it can work. And I debug in the pytorch-lightning , i found when loading data from validation dataloader it would be blocking. I am not sure what problem.
Environment:
docker; the lastest pytorch-lighting;gpu a100
log:
INFO Using validation DataLoader3DOffset with {}
INFO Building Sampling Cache for Dataloder
Sampling Cache: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1445.07it/s]
INFO Using 5 num_processes and 2 num_cached_per_queue for augmentation.                                                                                                           | 0/2 [00:00<?, ?it/s]
INFO VALIDATION KEYS:
odict_keys(['case_0', 'case_7'])
using pin_memory on device 0

I test the validation step and it world jam
can you help me, thank you !!!!!
sorry,  It is my fault. now i solve this problem