scaling: use simple ints for small counters and int32_t for scales and coeffs

This fixes a gcc warning and mostly makes the code more readable and avoids
some (int) casts
scaling: remove m_refMatrixId; it was only used to return an int

Only one value of the matrix was ever used at a time, so we can return thtat
index directly from checkPredMode
slicetype: compute frame satd cost for zero latency
search: separate bSkipRes == true path