I thought so too but the current implementation of the inverse_transform and possibly other functions do things very inefficiently if the matrix is not csr and they don't convert to csr. For Ex in inverse_transform:
```python
return [inverse_vocabulary[X[i, :].nonzero()[1]].ravel()
                for i in range(n_samples)]
```