Re #6191. Recover performance while not using SpectraDetectorMap

The previous change (ce9afb2) caused a x100 slowdown in the MaskDetectorsInShape
performance test. The changes here bring things back to where they were.