Properly calculate & store neighbor poiners