port: postpone peer delay calculation until nrr is valid

Signed-off-by: Delio Brignoli <dbrignoli@audioscience.com>