I think the check may be faster with NumPy's set operations:

    non_binary = np.setdiff1d(X, [0, 1])

then check whether this has length 0. If not, the value of `non_binary` can be reported in the error message.

Also the sparse and dense checks can be rolled into one:

    whatever_the_final_check_is(X.data if issparse(X) else X)