Use linear algorithm for add()/remove() performance (we only have small bitmaps), with shortcut for simple cases.