Re #6191. Recover performance while not using SpectraDetectorMap

The previous change (ce9afb2) caused a x100 slowdown in the MaskDetectorsInShape
performance test. The changes here bring things back to where they were.
Merge branch 'feature/6191_remove_spectradetectormap' into develop