remove some caching

this caching doesn't increase performance, but does increase complexity.
remove it for now and find better ways to speed up this code.