Uses the answers, not the labels. Fixes #284.