Properly handle early stopping from loss of multimappers