More speed ups. Determine the neighbors only once.