vec: fix compile warnings with clang at 16bpp - remove dead functions
test: bump pixel depth to 12 for HIGH_BIT_DEPTH builds