Test that frame values do not exceed bit depth