primitives: add support for 16-bit psyCost calculations
psy-rd: modify psy-rd cost decisions in search::xEstimateResidualQT

This function looks to minimizing rate/distortion in the "residual". Rate is bits
taken to encode residual, and distortion is ssd(original residual, reconstructed
residual). It is wrong to calculate psyCost with psy-energy = E(src, recon) here.
psyCost is now modified as E(orig residual, recon residual).