smooths prob stack using similarity transform of rigid model