Spot the Difference: Detection of Topological Changes via Geometric Alignment

1 Introduction . Geometric alignment is a fundamental component of widely different algorithms , ranging from domain adaptation [ 7 ] , optimal transport [ 40 ] and normalizing flows [ 35 , 42 ] in machine learning ; optical flow [ 21 , 51 ] and learned augmentation [ 20 ] in computer vision , and deformable registration within biomedical imaging [ 5 , 15 , 19 , 39 , 53 ] . A recurring challenge is the alignment of domains whose topology is not the same . When the objects to be aligned are probability distributions [ 35 ] , this appears when distributions have different numbers of modes whose support is separated into separate connected components . When the objects to be aligned are scenes or natural images , the problem occurs with occlusion or temporal changes [ 51 ] . In biomedical image registration , the problem is very common and happens when the studied anatomy differs from `` standard '' anatomy [ 36 ] . Despite being extremely common , this problem is routinely ignored or accepted as inevitable , potentially introducing bias in downstream analysis . We study two cases from biomedical image registration . One is the alignment of image slices to reconstruct a 3d volume , where changes in topology between slices introduce challenges in postprocessing ( Figure 1 ) . The other is the registration of brain MRI scans , where tumors give common examples of anatomies that are topologically different from healthy brains . In deformable image registration , a `` moving image '' is mapped via a nonlinear transformation to make it as similar as possible to a `` target '' image , enabling matching local features or transferring information from one 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . image to another . It is common to numerically stabilize the estimation of the transformation by constraining the predicted transformation to be diffeomorphic , that is , bijective and continuously differentiable in both directions . In particular , diffeomorphic transformations are homeomorphic , or topology-preserving , which implies that a common topology is assumed across all images [ 13 , 15 ] . This topology is often provided by a common template image Itemplate , from which all other images are obtained via the transformation Φ from the group of diffeomorphisms G. Under this common topology assumption , the set of all images is given by I = { Itemplate ◦ Φ|Φ ∈ G } . Topological differences in biomedical images can be caused by a variety of processes . For instance , image slices obtained from a volume do not all contain the same elements . Tumor growth or the removal of surgical tissue can alter the topology of an image . Various processes can lead to the replacement or deformation of organic tissue , which can not be mapped to the original image . We choose to model these topological differences as the inability to obtain one image from the other via a homeomorphic transformation of the image domain . Since , within image registration , transformations are assumed to be continuously differentiable , we are effectively modelling topological differences between pairs of images via the failures of diffeomorphic image registration in aligning them . As most registration algorithms align images based on intensity , e.g . minimizing mean squared error ( MSE ) , these tissue changes make it difficult to map images correctly . The strong local deformations required to deal with the non-diffeomorphic part of the image inevitably also deform the surrounding area , leading to distorted transformation fields in topologically matching parts of the image [ 36 ] . These transformation fields adversely affect downstream tasks , for example indicating false size changes in adjacent regions . Previous work on aligning topologically inconsistent domains . Attempting to relax the sameimage assumption induced by fully diffeomorphic transformations is not new . In the context of organs sliding against each other , several approaches exist , most of which rely on pre-annotating the sliding boundary using organ segmentation [ 6 , 10 , 22 , 37 , 43 , 46 ] , with a few extensions to un-annotated images [ 38 , 45 ] . When topological holes are created or removed in the domain , for example through tumors , pathologies , or surgical resections , the loss function used for registration can be locally weighted or masked [ 26 , 29 , 30 ] , or an artificial insection can be grown to correct anatomies [ 36 ] . These approaches rely on annotation of the topological differences , which have to be provided manually or by segmentation . An exception is given by Li and Wyatt [ 30 ] , which detects changes in topology from the difference between the aligned images . This depends crucially on the ability to find a good diffeomorphic registration outside the anomaly , which is difficult all the while the applied transformation is still diffeomorphic . An alternative approach to registering topologically inconsistent images is to inpaint the difference in the source images to obtain a topologically consistent quasi-normal image . Then standard registration methods can be used on the altered images . Quasi-normal images can be obtained through lowrank and sparse matrix decomposition [ 32 , 33 ] , principle component analysis [ 16 , 18 ] , denoising VAEs [ 52 ] , or learning of a blended representation [ 17 ] . Registration with the quasi-normal approach retains the diffeomorphic properties of the transformation but does not register the topologically inconsistent areas of the images . Our contribution . We propose an unsupervised algorithm for the detection of changes in image topology . To this end , we train a conditional variational autoencoder for predicting image-to-image alignment , obtaining a per-target-pixel probability of being obtained from the moving image via diffeomorphic transformation . We combine a semantic loss function trained to extract contextual information [ 8 ] , with a learnable prior of transformations [ 9 ] , allowing us to incorporate both the reconstruction error , as well as knowledge about the expected transformation strength . We test the validity of our approach on a novel dataset of cell slices with annotated topological changes and on the proxy task of unsupervised brain-tumor detection . We also validate our approach by investigating a spatial `` topological inconsistency likelihood '' , and showing that this likelihood is higher in regions where topological inconsistencies are known to be common . Our model is able to detect topological inconsistencies with a purely registration-driven framework , and thus provides the first step towards an end-to-end registration model for images with topological discrepancies . The implementation is available at github.com/SteffenCzolbe/TopologicalChangeDetection . 2 Background . 2.1 Notation of images and transformations . We view an image I interchangeably as two different structures . First , it is a continuous function I : ΩI → R C , where ΩI = [ 0 , 1 ] D is the domain of the image , and C the number of channels . This function can be approximated by a grid of n pixels with positions xk ∈ ΩI leading to the image representation I ( c ) k , where c is an index over the channels and Ik = ( I ( 1 ) k , . . . , I ( C ) k ) T = I ( xk ) . Second , this pixel grid is accompanied by a graph structure that encodes the neighbourhood of each pixel . In this view , the set of neighbours of a pixel with index k ( for example the 4-neighbourhood of a pixel on the image grid ) is referred to as N ( k ) and |N ( k ) | is the number of neighbours . The neighborhoods of a pixel gives rise to a graph which can be described via the graph laplacian Λ ∈ Rn×n with Λk , k = |N ( k ) | and Λk , k′ = −1 when pixel k ′ ∈ N ( k ) , and zero otherwise . Applying a spatial transformation Φ : RD → RD to an image is written as J = I ◦ Φ , which can be seen as its own image with domain ΩJ = [ 0 , 1 ] D with pixel coordinates yk ∈ ΩJ and Jk = I ( Φ ( yk ) ) . The transformation Φ can be seen as a vector field on the image domain which assigns each pixel in J a position on I and thus it can be parameterized as a pixel grid Φ ( d ) k , d = 1 , . . . , D at the pixel coordinates of J using Φ ( yk ) = yk +Φk . To make this choice of coordinate system clear , we will refer to a transformation that moves a pixel position from the domain ΩJ to the corresponding pixel in domain ΩI as ΦJ→I , whenever it is not clear from the context . If Φ is a diffeomorphism , it can alternatively be parameterized by a vector field V on the tangent space around the identity , where the mapping between the tangent space and the transformation is given by Φ = exp ( V ) , which amounts to integration over the vector field [ 2 ] . 2.2 Variational registration framework . It is possible to phrase the problem of fitting a registration model in terms of variational inference , using an approach similar to conditional variational autoencoders [ 47 ] . Here , we summarize the approach taken by [ 9 , 31 ] . For a D-dimensional image pair ( I , J ) , we assume that J is generated from I by drawing a transformation Φ from a prior distribution p ( Φ|I ) , apply it to I and then add pixel-wise noise : p ( J|I ) = ∫ pnoise ( J|I ◦ Φ ) p ( Φ|I ) dΦ This includes the common topology assumption implicitly via p ( Φ|I ) , which is typically chosen to produce invertible transformations depending only on the topology of I , as well as the noise model which does not assume systematic changes between J and I . This model can be learned using variational inference using a proposal distribution q ( Φ|I , J ) with evidence lower bound ( ELBO ) log p ( J|I ) ≥ Eq ( Φ|I , J ) [ log pnoise ( J|I ◦ Φ ) ] −KL ( q ( Φ|I , J ) ‖p ( Φ|I ) ) . ( 1 ) In contrast to variational autoencoders , the decoder is given by the known application of Φ to I . Thus , the degrees of freedom in this model are in the choice of the encoder , prior , and the noise distribution . Dalca et al . [ 9 ] proposed to parameterize Φ as a vector field V ( d ) k on the tangent space , which turns application of Φ = exp ( V ) into sampling an image with a spatial transformer module [ 24 ] . As a prior for this parameterization , they chose a prior independent of I p ( Φ ) = D ∏ d=1 N ( V ( d ) | 0 , Λ−1 ) , where we used the implicit identification of Φ and V and the precision matrix Λ is chosen as the Graph Laplacian over the neighbourhood graph ( see notation ) . Using an encoder that for each pixel proposes q ( V ( d ) k |I , J ) = N ( µ ( d ) k , v ( d ) k ) , the KL divergence is derived as KL ( q ( Φ|I , J ) ‖p ( Φ|I ) ) = 1 2 D ∑ d=1 n ∑ k=1 − log v ( d ) k +|N ( k ) |v ( d ) k + ∑ l∈N ( k ) ( µ ( d ) k − µ ( d ) l ) 2 +const . ( 2 ) It is worth noting that this equation is invariant under translations of µ . This invariance manifests in rank-deficiency of Λ and as a result , const is infinite . Thus , sampling from the prior and bounding the objective is impossible . Still training with this term works in practice as images are usually pre-aligned with an affine transformation and thus translations are close to zero . We will present a slightly modified approach , rectifying the missing eigenvalue .

Summary: In this paper, the authors present an unsupervised topology difference detector based on geometric alignment. The method is based on a conditional variational autoencoder. It allows pixel-wise outlier detection. The method is evaluated on the task of brain tumor detection and compared to other unsupervised segmentation/registration methods and a supervised segmentation baseline. The results show that the presented method performs best compared to other unsupervised segmentation/registration methods.