fail if binary_col_modifier can't find all involved columns.