Remove redundant indices, data, indptr appends in predict

Take them out of the four branches and do them once at the end of
all of the conditionals