level: signal profile/level NONE if non-compliant configuration is found

Once you get above a certain resolution, the CTU size must be at least 32. There
is no level that supports a smaller CTU, so signal level NONE.  It's my
understanding that you cannot signal a profile if the level is NONE, so we
reset profile to NONE as well.

The same is true if NumPocTotalCurr is greater than 8; there are no levels which
support values that high.  Instead of signaling the closest level, we should
signal profile/level NONE
rc: nits; add LF in frame stats of 2 pass
quant: remove scaledCoeff from nquant()