Yeah it should be on Softmax, I missed that this was the projection layer. Now that you mentioned it, would we save memory if change Softmax's output_space to IndexSpace, and use pylearn2.format.target_format.OneHotFormatter in Softmax.cost?