Fix bug that didn't remove blocks from the field and thus allowed selecting empty fields