I don't think user is part of the polymorphic association with votes. The User to Votes association is a one-to-many. 
User
    has_many :votes
Vote
    belongs_to :user