Optimize highlight references

A few further optimizations based on @DanielFabian 's profile information.