[WIP] remove check_arrays stuff and old input validation

Follow-up on #3443, removing all the old stuff.
I think it's better to leave at is it as in cross-decomposition module, X and Y are fundamentally symmetric, so better reflect that explicitly in the input validation.