Repository: seung-lab/cloud-volume
Pull request #482 (2021-06-26 04:39:50)

fix: update morton coding
For grid dimensions that are a power of 2, the code was attributing an extra bit.  Fixing that also seems to fix the prior problem of size=1 dimensions (a power of 2!) naturally, without the extra check.

I added a test for the case I recently hit, where the morton code was out of range (195, where the #chunks was 128).  In addition, a prior test was incorrect and updated.

chrisroat: @william-silversmith If you are creating a new igneous docker image, can you include this fix, as well?
william-silversmith: Thanks Chris! I'll need to think about this, but I can say that if my code is wrong so is the spec. (!)
https://github.com/google/neuroglancer/blob/master/src/neuroglancer/datasource/precomputed/volume.md#compressed-morton-code
chrisroat: No worries.  I just rolled my own igneous image.

Yeah, I had to think about it a while, too.  I was getting some nasty re-arragements of my data (shard-sized chunks in the wrong place, or entirely missing) when running downsampling, and this seems to fix them.

Patch:
diff --git a/cloudvolume/datasource/precomputed/image/common.py b/cloudvolume/datasource/precomputed/image/common.py
index 11c8c49..07856ef 100644
--- a/cloudvolume/datasource/precomputed/image/common.py
+++ b/cloudvolume/datasource/precomputed/image/common.py
@@ -88,7 +88,7 @@ def compressed_morton_code(gridpt, grid_size):
 
   for i in range(max(num_bits)):
     for dim in range(3):
-      if 2 ** i <= grid_size[dim] and grid_size[dim] > 1:
+      if 2 ** i < grid_size[dim]:
         bit = (((np.uint64(gridpt[:, dim]) >> np.uint64(i)) & one) << j)
         code |= bit
         j += one
@@ -123,4 +123,3 @@ def shade(dest_img, dest_bbox, src_img, src_bbox):
     src_img = src_img[..., np.newaxis]
   
   dest_img[ dbox.to_slices() ] = src_img[ sbox.to_slices() ]
-
diff --git a/cloudvolume/datasource/precomputed/metadata.py b/cloudvolume/datasource/precomputed/metadata.py
index 9ba5903..efa2251 100644
--- a/cloudvolume/datasource/precomputed/metadata.py
+++ b/cloudvolume/datasource/precomputed/metadata.py
@@ -127,7 +127,8 @@ class PrecomputedMetadata(object):
       info['redirect'] = str(redirect)
  
     # add mip levels
-    for mip in range(max_mip):
+    # the max_mip should be inclusive
+    for mip in range(max_mip + 1):
       cls.add_scale(None, factor ** mip, encoding=encoding, chunk_size=chunk_size, info=info)
 
     if encoding == 'compressed_segmentation':


Test patch:
diff --git a/test/test_sharding.py b/test/test_sharding.py
index 456f493..54ee89b 100644
--- a/test/test_sharding.py
+++ b/test/test_sharding.py
@@ -69,13 +69,16 @@ def test_compressed_morton_code():
     assert False
   except ValueError:
     pass
-  assert cmc((1,2,0)) == 0b001001
+  assert cmc((1,2,0)) == 0b000101
 
-  assert np.array_equal(cmc([(0,0,0), (1,2,0)]), [0b000000, 0b001001])
+  assert np.array_equal(cmc([(0,0,0), (1,2,0)]), [0b000000, 0b000101])
 
   cmc = lambda coord: compressed_morton_code(coord, grid_size=(4,4,1))
   assert cmc((3,3,0)) == 0b1111
 
+  cmc = lambda coord: compressed_morton_code(coord, grid_size=(8,8,2))
+  assert cmc((5,5,0)) == 0b1100011
+
 def test_image_sharding_hash():
   spec = ShardingSpecification(
     type="neuroglancer_uint64_sharded_v1",
@@ -265,5 +268,3 @@ def test_write_image_shard():
     assert False
   except exceptions.AlignmentError:
     pass
-
-