FIX: Don't copy X at imputation in the sparse case