Hardcode the neighbors assignment.

The old way wasn't really saving that much code, and this function is dominating instruments.