> If sklearn or something else has some functions for automatically
returning the neighbors in a 3D grid, then we could use that. This has the
advantage of giving the exact same list, in the exact same order, as the C
code, which is nice

maybe :

sklearn.feature_extraction.image.grid_to_graph

can help.