Do you really need to create that `X_content` matrix? It seems to me that you are always using the columns separately. I think it might be possible to save some copies if you use separate variables instead (in particular, in the sparse case).