Gather score from alll direct neighbors