Spot the Difference: Detection of Topological Changes via Geometric Alignment

1 Introduction . Geometric alignment is a fundamental component of widely different algorithms , ranging from domain adaptation [ 7 ] , optimal transport [ 40 ] and normalizing flows [ 35 , 42 ] in machine learning ; optical flow [ 21 , 51 ] and learned augmentation [ 20 ] in computer vision , and deformable registration within biomedical imaging [ 5 , 15 , 19 , 39 , 53 ] . A recurring challenge is the alignment of domains whose topology is not the same . When the objects to be aligned are probability distributions [ 35 ] , this appears when distributions have different numbers of modes whose support is separated into separate connected components . When the objects to be aligned are scenes or natural images , the problem occurs with occlusion or temporal changes [ 51 ] . In biomedical image registration , the problem is very common and happens when the studied anatomy differs from `` standard '' anatomy [ 36 ] . Despite being extremely common , this problem is routinely ignored or accepted as inevitable , potentially introducing bias in downstream analysis . We study two cases from biomedical image registration . One is the alignment of image slices to reconstruct a 3d volume , where changes in topology between slices introduce challenges in postprocessing ( Figure 1 ) . The other is the registration of brain MRI scans , where tumors give common examples of anatomies that are topologically different from healthy brains . In deformable image registration , a `` moving image '' is mapped via a nonlinear transformation to make it as similar as possible to a `` target '' image , enabling matching local features or transferring information from one 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . image to another . It is common to numerically stabilize the estimation of the transformation by constraining the predicted transformation to be diffeomorphic , that is , bijective and continuously differentiable in both directions . In particular , diffeomorphic transformations are homeomorphic , or topology-preserving , which implies that a common topology is assumed across all images [ 13 , 15 ] . This topology is often provided by a common template image Itemplate , from which all other images are obtained via the transformation Φ from the group of diffeomorphisms G. Under this common topology assumption , the set of all images is given by I = { Itemplate ◦ Φ|Φ ∈ G } . Topological differences in biomedical images can be caused by a variety of processes . For instance , image slices obtained from a volume do not all contain the same elements . Tumor growth or the removal of surgical tissue can alter the topology of an image . Various processes can lead to the replacement or deformation of organic tissue , which can not be mapped to the original image . We choose to model these topological differences as the inability to obtain one image from the other via a homeomorphic transformation of the image domain . Since , within image registration , transformations are assumed to be continuously differentiable , we are effectively modelling topological differences between pairs of images via the failures of diffeomorphic image registration in aligning them . As most registration algorithms align images based on intensity , e.g . minimizing mean squared error ( MSE ) , these tissue changes make it difficult to map images correctly . The strong local deformations required to deal with the non-diffeomorphic part of the image inevitably also deform the surrounding area , leading to distorted transformation fields in topologically matching parts of the image [ 36 ] . These transformation fields adversely affect downstream tasks , for example indicating false size changes in adjacent regions . Previous work on aligning topologically inconsistent domains . Attempting to relax the sameimage assumption induced by fully diffeomorphic transformations is not new . In the context of organs sliding against each other , several approaches exist , most of which rely on pre-annotating the sliding boundary using organ segmentation [ 6 , 10 , 22 , 37 , 43 , 46 ] , with a few extensions to un-annotated images [ 38 , 45 ] . When topological holes are created or removed in the domain , for example through tumors , pathologies , or surgical resections , the loss function used for registration can be locally weighted or masked [ 26 , 29 , 30 ] , or an artificial insection can be grown to correct anatomies [ 36 ] . These approaches rely on annotation of the topological differences , which have to be provided manually or by segmentation . An exception is given by Li and Wyatt [ 30 ] , which detects changes in topology from the difference between the aligned images . This depends crucially on the ability to find a good diffeomorphic registration outside the anomaly , which is difficult all the while the applied transformation is still diffeomorphic . An alternative approach to registering topologically inconsistent images is to inpaint the difference in the source images to obtain a topologically consistent quasi-normal image . Then standard registration methods can be used on the altered images . Quasi-normal images can be obtained through lowrank and sparse matrix decomposition [ 32 , 33 ] , principle component analysis [ 16 , 18 ] , denoising VAEs [ 52 ] , or learning of a blended representation [ 17 ] . Registration with the quasi-normal approach retains the diffeomorphic properties of the transformation but does not register the topologically inconsistent areas of the images . Our contribution . We propose an unsupervised algorithm for the detection of changes in image topology . To this end , we train a conditional variational autoencoder for predicting image-to-image alignment , obtaining a per-target-pixel probability of being obtained from the moving image via diffeomorphic transformation . We combine a semantic loss function trained to extract contextual information [ 8 ] , with a learnable prior of transformations [ 9 ] , allowing us to incorporate both the reconstruction error , as well as knowledge about the expected transformation strength . We test the validity of our approach on a novel dataset of cell slices with annotated topological changes and on the proxy task of unsupervised brain-tumor detection . We also validate our approach by investigating a spatial `` topological inconsistency likelihood '' , and showing that this likelihood is higher in regions where topological inconsistencies are known to be common . Our model is able to detect topological inconsistencies with a purely registration-driven framework , and thus provides the first step towards an end-to-end registration model for images with topological discrepancies . The implementation is available at github.com/SteffenCzolbe/TopologicalChangeDetection . 2 Background . 2.1 Notation of images and transformations . We view an image I interchangeably as two different structures . First , it is a continuous function I : ΩI → R C , where ΩI = [ 0 , 1 ] D is the domain of the image , and C the number of channels . This function can be approximated by a grid of n pixels with positions xk ∈ ΩI leading to the image representation I ( c ) k , where c is an index over the channels and Ik = ( I ( 1 ) k , . . . , I ( C ) k ) T = I ( xk ) . Second , this pixel grid is accompanied by a graph structure that encodes the neighbourhood of each pixel . In this view , the set of neighbours of a pixel with index k ( for example the 4-neighbourhood of a pixel on the image grid ) is referred to as N ( k ) and |N ( k ) | is the number of neighbours . The neighborhoods of a pixel gives rise to a graph which can be described via the graph laplacian Λ ∈ Rn×n with Λk , k = |N ( k ) | and Λk , k′ = −1 when pixel k ′ ∈ N ( k ) , and zero otherwise . Applying a spatial transformation Φ : RD → RD to an image is written as J = I ◦ Φ , which can be seen as its own image with domain ΩJ = [ 0 , 1 ] D with pixel coordinates yk ∈ ΩJ and Jk = I ( Φ ( yk ) ) . The transformation Φ can be seen as a vector field on the image domain which assigns each pixel in J a position on I and thus it can be parameterized as a pixel grid Φ ( d ) k , d = 1 , . . . , D at the pixel coordinates of J using Φ ( yk ) = yk +Φk . To make this choice of coordinate system clear , we will refer to a transformation that moves a pixel position from the domain ΩJ to the corresponding pixel in domain ΩI as ΦJ→I , whenever it is not clear from the context . If Φ is a diffeomorphism , it can alternatively be parameterized by a vector field V on the tangent space around the identity , where the mapping between the tangent space and the transformation is given by Φ = exp ( V ) , which amounts to integration over the vector field [ 2 ] . 2.2 Variational registration framework . It is possible to phrase the problem of fitting a registration model in terms of variational inference , using an approach similar to conditional variational autoencoders [ 47 ] . Here , we summarize the approach taken by [ 9 , 31 ] . For a D-dimensional image pair ( I , J ) , we assume that J is generated from I by drawing a transformation Φ from a prior distribution p ( Φ|I ) , apply it to I and then add pixel-wise noise : p ( J|I ) = ∫ pnoise ( J|I ◦ Φ ) p ( Φ|I ) dΦ This includes the common topology assumption implicitly via p ( Φ|I ) , which is typically chosen to produce invertible transformations depending only on the topology of I , as well as the noise model which does not assume systematic changes between J and I . This model can be learned using variational inference using a proposal distribution q ( Φ|I , J ) with evidence lower bound ( ELBO ) log p ( J|I ) ≥ Eq ( Φ|I , J ) [ log pnoise ( J|I ◦ Φ ) ] −KL ( q ( Φ|I , J ) ‖p ( Φ|I ) ) . ( 1 ) In contrast to variational autoencoders , the decoder is given by the known application of Φ to I . Thus , the degrees of freedom in this model are in the choice of the encoder , prior , and the noise distribution . Dalca et al . [ 9 ] proposed to parameterize Φ as a vector field V ( d ) k on the tangent space , which turns application of Φ = exp ( V ) into sampling an image with a spatial transformer module [ 24 ] . As a prior for this parameterization , they chose a prior independent of I p ( Φ ) = D ∏ d=1 N ( V ( d ) | 0 , Λ−1 ) , where we used the implicit identification of Φ and V and the precision matrix Λ is chosen as the Graph Laplacian over the neighbourhood graph ( see notation ) . Using an encoder that for each pixel proposes q ( V ( d ) k |I , J ) = N ( µ ( d ) k , v ( d ) k ) , the KL divergence is derived as KL ( q ( Φ|I , J ) ‖p ( Φ|I ) ) = 1 2 D ∑ d=1 n ∑ k=1 − log v ( d ) k +|N ( k ) |v ( d ) k + ∑ l∈N ( k ) ( µ ( d ) k − µ ( d ) l ) 2 +const . ( 2 ) It is worth noting that this equation is invariant under translations of µ . This invariance manifests in rank-deficiency of Λ and as a result , const is infinite . Thus , sampling from the prior and bounding the objective is impossible . Still training with this term works in practice as images are usually pre-aligned with an affine transformation and thus translations are close to zero . We will present a slightly modified approach , rectifying the missing eigenvalue .

Summary: This paper proposes a novel unsupervised topological difference detection algorithm. The model is based on a conditional variational auto-encoder and detects topological anomalies with regards to a reference alongside the registration step. In terms of the experiments, as there exists no standard dataset of annotated topological differences for image registration, this paper validates the proposed topological difference detection algorithm on a proxy task of unsupervised anomaly detection in images.