Optimize RAM usage & speed up 7 times faster

Instead of O(dpm^2), it is now O(nThread * d * m^2)