Make frame rendering much more efficient. (Part of the TODO)

This might slightly break rendering but we'll see.