many optimizations.  also implemented a ram cache to avoid recaculating digests for the same data streams