The loop for finding the correct file and segment is the same as Obj::linnum. As functions are usually emitted as COMDAT into their own segment, the line numbers should only cover the current function. I don't think that the line numbers can be assumed to be sorted, can they?
It would be even better if the codegen records the corresponding code offset into the symbols. Is that possible?

I just measured build the "quick" dmd testsuite (I always run it with -g): master took 1min58, with this PR, it took 1min59.