Calculate order overlap for given level of off-band rejection