lightmap: Don't iterate unless you need to