[overlay][fix] fix rescaling when coordinate-differences are larger than recaling range,
which is unfortunately slipped through. This fixes ticket #10747