[MRG] FIX: Remove lil matrix conversion

Removed a XXX from the code. (unnecessary LIL conversion)
Hmm. I tried something similar to this, but I got a segmentation fault. I shall have a look again.