Remove optional input validation from input_validation

Manually removed `check_X_y` since there was still some useful stuff related to repeated typecasting in check_pairwise. Fixes https://github.com/scikit-learn/scikit-learn/issues/3963 . ping @larsmans