I think we should do precomputation of label-fields in attachToForm function because if we do it in isValid every time https://github.com/rmcdaniel/joomla-cms/blob/d2c5e2c27f403bb3629ff4f4c76c53e1b1dd75b3/media/system/js/validate-uncompressed.js#L79 called we compute the same labels again and again.

I did it in https://github.com/Achal-Aggarwal/joomla-cms/compare/jquery-validation