Memoize neighbors