Optimize highlight references

A few further optimizations based on @DanielFabian 's profiling reports.