Treat incorrect declarations as errors in VS.
Fix bug in search relating to transform with no non-zero coefficients.

- Because cost was calculated even though there were no coefficients, these
  very good modes were less likely to be selected.

- Added assert to encode_coeff_nxn to avoid these problems in the future.
Give correct transform depth to RDOQ.

Conflicts:
	src/search.c