Avoid requesting precached votes for unvoteable things

Looks like this code got a little wonky when it got reorganized in
de8c02167a208bbb918eb2bbc8510b3d26063404 , after that commit
precached votes started being fetched even when a builder passed
in a list made up entirely of unvoteable things.