Got it!
I'm an idiot, I was making a slice with index 0,1,2 and then appending the guess values to that, which made their indexes 3,4,5 which were never used.

Removed the appending, just use the indexes that are created.  We are hard coding this to 3 dimensions right now anyway.