[WIP] remove check_arrays stuff and old input validation

Follow-up on #3443, removing all the old stuff.
check_X_y?