Convert the column index into the indices in each of the Variable index sets. It's ugly.