don't release color buffers when they are still in use

Fixes most of #38.