Resolution and Field of View Invariant Generative Modelling with Latent Diffusion Models

Authors: ashay  patel
mark s graham
vicky  goh
sebastien  ourselin
m jorge cardoso
Venue: MIDL_io_2024_Conference
Published: 2024-01-25 15:58:56

Abstract: Large dataset requirements for deep learning methods can pose a challenge in the medical field, where datasets tend to be relatively small. Synthetic data can provide a suitable solution to this problem, when complemented with real data. However current generative methods normally require all data to be of the same resolution and, ideally, aligned to an atlas. This not only creates more stringent restrictions on the training data but also limits what data can be used for inference. To overcome this our work proposes a latent diffusion model that is able to control sample geometries by varying their resolution, field of view, and orientation. We demonstrate this work on whole body CT data, using a spatial conditioning mechanism. We showcase how our model provides samples as good as an ordinary latent diffusion model trained fully on whole body single resolution data. This is in addition to the benefit of further control over resolution, field of view, orientation, and even the emergent behaviour of super-resolution. We found that our model could create realistic images across the varying tasks showcasing the potential of this application.

Main paper: 
 introduction: 
 Deep learning methods have achieved remarkable growth in recent years in both natural imaging and medical imaging analysis. Nevertheless deep learning methods are data-hungry, and the size of publicly available medical datasets are much smaller than their natural image counterparts. These sizes can be drastically smaller depending on the area of focus, with the average dataset being less than 200 samples (Kiryati and Landau, 2021) . This limitation is also increased by the stringent data requirements for medical imaging. Most work requires data to be of the same resolution aligned to an atlas, in addition to showcasing the same field of view. This can consequently require large amounts of preprocessing and/or simplified model design which can result in poor generalisation (Varoquaux and Cheplygina, 2022; Decuyper et al., 2021; Dinsdale et al., 2022) . A restriction we hope to overcome with our proposed model invariant to such characteristics, as displayed in figure 1 .Being able to produce high quality synthetic data can provide the medical imaging community a promising alternative to improve ones ability to carry out the appropriate Figure 1 : Spatial Conditioning provided by generating any arbitrary affine transform grid that showcases the mapping from a group space grid to a new space that defines the images orientation, resolution and field of view. The coordinate grid is then concatenated to the input fed into the diffusion model to control sampling. The three paths show a coarse affine coordinate grid to generate a low resolution image (blue), a fine grained grid for high resolution image (green) and a rotated fine grained image to create a rotated high resolution image (red).research on larger scales (Jordon et al., 2020 (Jordon et al., , 2022;; Wang et al., 2021) . The use of generative models have shown a promising solution for this, by learning the probability density function of the underlying training data, which they can then use to create new realistic examples.Variational Autoencoders (VAEs) (Baur et al., 2020) are a simple baseline for generative modelling in imaging, however they have been shown to produce blurry samples with a number of better alternatives. Until recently, Generative Adversarial Networks (GANs) have been at the forefront of generative modelling research (Creswell et al., 2017; Kwon et al., 2019; Wang et al., 2020) . However an added difficulty within medical imaging, especially with 3D images, is the computational requirements. Further limitations of such networks are commonly documented drawbacks, namely unstable training, mode collapse and failure to converge (Kodali et al., 2017) . In addition they are hard to guide via conditioning.Recently diffusion models have shown great promise and development showing stateof-the-art results for generating synthetic images, surpassing GANs in sample quality and diversity (Dhariwal and Nichol, 2021) . They have also been shown to effectively create images based on various conditionings i.e. class-based, image and text-based (using techniques like classifier-free guidance (Ho and Salimans, 2022) ). Notable real-life examples include Latent Diffusion Models (LDMs) (Rombach et al., 2021) and DALL-E (Ramesh et al., 2021) . Latent diffusion models specifically have shown promise in medical imaging as they make use of an Autoencoder model to compress data, and then a diffusion model to work on the compressed latent space. The work in (Pinaya et al., 2022) and (Graham et al., 2023) showcases the use of LDMs in medical imaging for generative modelling and out-of-distribution detection respectively displaying state-of-the-art results.Even so, current methods in generative modelling lack in diversity of image geometry. In medical imaging, they are all designed to generate images at a particular dimension, resolution and field-of-view. Models able to train and infer on varying data types be it resolution and field-of-view is still a relatively new concept, however as demonstrated in (Patel et al., 2023) , they can have a number of benefits including maximising data utilisation, reducing data preprocessing and improving generalisability. As such being able to compliment such a model and producing a generative model with the added flexibility of being able to control the resolution, field-of-view and orientation can have significant benefits to further pursuing the goal of a singular model with performance capabilities not limited to single group space.To do so we should be modelling both the image and the affine mapping to a group space.In this study we aim to showcase the use of a Latent Diffusion Model (LDM) with a spatial conditioning that defines both the affine transform grid to a group space and voxel sizing of an image that allows us to control the resolution of images generated, in additional to the field-of-view and orientation. We demonstrate this work on a dataset of 1014 3D whole body CT scans. We showcase that the addition of this spatial conditioning still generates samples with comparable quality to an LDM trained only on whole body images of one resolution, but with the added benefit of control over the generation. Additionally, without specifically training for such a task, we show a unique emergent behaviour by this model for the downstream task of image super-resolution to any specified resolution. 
generative modelling: 
 Our methodology is based around LDMs, which use an autoencoder-based model to compress input data into a lower-dimensional latent representation in conjunction with the generative modelling properties of a diffusion model. As we are working with high-dimensional 3D data, the compression model is an essential step. For this we trained a Vector Quantized-Variational Autoencoder (van den Oord et al., 2017) .Vector Quantized-Variational Autoencoder: The VQ-VAE consists of an encoder part that maps an image x ∈ R H×W ×D onto a compressed latent representation z ∈ R h×w×d×nz where n z is the latent embedding vector dimension. The compressed latent representation z is then passed through a quantization block where each feature column vector is mapped to its nearest codebook vector via a euclidean distance metric. Each spatial code z ijl ∈ R nz is then replaced by its nearest codebook element e k ∈ R nz , k ∈ 1, ..., K where K denotes the codebook vocabulary size, resulting in the quantized latent space z q . Given z q , the VQ-VAE decoder then reconstructs the observations x ∈ R H×W ×D . In practice we employ a VQ-GAN model (Esser et al., 2020) , to produce higher quality reconstructions by employing adversarial training. See appendix B for implementation details.Denoising Diffusion Probabilistic Models: Once we have trained the VQ-GAN, we can then train a Denoising Diffusion Probablistic Model (DDPM) (Ho et al., 2020) on the sample latents (de-quantized latents). During training, noise is added to the latent z with respect to a timestep t using a fixed Gaussian noise schedule defined by β t rendering a noised sample z t such that:q(z t |z 0 ) = N z t √ ᾱt z 0 , (1 -ᾱt ) I (1)where z 0 refers to a noise free latent z and 0 < t ≤ T . Additionally α t := 1 -β t and ᾱt := t s=1 α s . The aim is to generate a network that can perform the reverse process or denoising process which can be presented as the Gaussian transition:EQUATIONFrom (Ho et al., 2020) we can train our network ϵ θ (z t , t) to predict the noise directly in the forward noising process, ϵ. We can then train with the simplified objective loss L(θ) = E t,z 0 ,ϵ ∥ϵ -ϵ θ (z t )∥ 2 and denoise according to :EQUATIONwhere n ∼ N (0, I) and σ t 2 = β t For training, we use a noise schedule of T = 1000 with a scaled linear variance noise schedule with β 0 = 0.0015 and β T = 0.0195. See appendix C for implementation details.As the VQ-GAN and DDPM are purely convolution, we are able to input data of varying dimension i.e. allowing for data at varying resolution and FOVs. However this requires a form of spatial conditioning that enables modelling of such geometric variations. To produce images of varying resolution, FOV and orientation we need to model both the image and the geometry of said image represented by the affine mapping of the given image to a group space plus the grid size (or voxel dimensions). This affine transformation gives us an explicit coordinate frame to work from to control the image generation.VQ-VAE Spatial Conditioning: To control the generation we require a form of conditioning to help the VQ-VAE encode and reconstruct data at varying resolutions and also to help steer the denoising process in the DDPM. For this we take inspiration from the work in CoordConv (Liu et al., 2018) . This is further adapted to model more intricate image geometries to essentially define any affine transform from one image to a group space, as similarly explored in (Patel et al., 2023) for anomaly detection. This accounts for some level of spatial awareness in the data fed into the models. A CoordConv layer is a concatenation of channels to the input image referencing a predefined coordinate system. After concatenation, the input can then be fed through any given convolutional layer as normal. For a 3D image, we would thus have 3 coordinates, ijk, where the i coordinate channel is a h × w × d rank-1 matrix with its first row filled with 0's, second row with 1's and so on. This would then of course be the same for the j coordinate channel, but with columns filled with constant values not rows, and likewise for the k channel in a depth-wise fashion. These channels would then be normalised between [0,1]. The main utilisation of the coordinate system in this work however is due to the 0-1 scale across the channels regardless of image resolution. For example 2 whole body images of varied resolution and dimensions would still have coordinate channels ranging from 0-1 along each axis, but with differing increments between each voxel, thus conveying the notion of spatial resolution i.e. showcasing the grid sizing of the affine. It was found in (Patel et al., 2023) , that training a VQ-VAE on images with large variations in resolution resulted in poor reconstructions, and the coordinate channels helped to overcome this issue. Furthermore, we can further adapt these coordinates to convey a notion of the field-of-view (FOV) present in the image. It is common when working with various types of scans of the body, that a whole-body scan is not always taken. This data can still be useful if we can provide a way for our models to work with data of varying FOVs. We can convey this by adapting the 0-1 range of the coordinate channels. In our case a whole body image (upper leg to neck in this work) represents our full range of [0,1], where in the axial plane 0 represents the bottom of the body and 1 represents the top. We then contract the range to represent a different FOV, i.e. perhaps [0.5,1] to showcase a scan of the upper body only. A graphic demonstration of this can be found in Figure 2 also highlighting changes to the coordinate grid as a result of a rotation. Note that while the proposed model assumes translation, scale and rotational changes, as we are modelling the affine transform to a group space, this can be trivially extended to a full affine mapping of the coordinate system including shearing. LDM Spatial Conditioning: To include the spatial conditioning in the DDPM, the coordinate channels used in the VQ-GAN encoding, are downsampled to the size of the latent space via an average pooling with kernel size and stride of 8 , equaivalent to the downsampling applied in the VQ-GAN. This then gives us 3 spatial channels at the latent dimension that define the affine and grid sizing in order to steer the generation process. These channels are then concatenated as channels to the input to condition the generation. For sampling, we define the extremes of the coordinate channels and desired dimension, for which we can then linearly interpolate the intermediate values to give the conditioning.Further to the generative modelling capabilities of our model and as a result of our model being able to control generation at varying resolutions, we showcase an emergent behaviour of our model -super-resolution. This requires no further training and is purely an emergent behaviour of the training mechanism and conditioning employed in this methodology. As reported in the work of (Ho et al., 2020) , different t values are responsible for modelling different image features, with higher values associated with coarser features, and lower t values responsible for finer details. We can take this information to define a suitable denoising schedule to super-resolve an image. The method works by taking a low resolution image, and upscaling it via a linear upscaling to the desired dimension we require. We then encode this to get our latent space. At which point we partially noise the encoded image and then proceed to denoise the image. Utilising the information that finer grained detailed occur at lower t-values, and that the larger scale details are already present from the original upscaled image we use a low t value of t = 100 chosen empirically from visualising samples qualitatively. It was found that higher t values eventually resulted in larger changes in the structure of the CT image away from the original sample, although sample quality was higher. To overcome this we repeated the noising and denoising process. The advantage of this was to slowly improve the image resolution over cycles without deviating from the original CT content. The final process chosen made use of 5 cycles of noising the image to t= 100 and denoising. For this we use a Denoising Diffusion Implicit Model Scheduler which does not add any noise during each denoising step to prevent the image moving away from the original sample. One highlight of this emergent behaviour is that we did not require any paired low and high resolution images, and additionally we are able to super-resolve from any arbitrary resolution to any higher resolution. Our first experiment is designed to demonstrate the generative modelling capabilities of our method for both whole-body CT images at a fixed intermediate resolution in addition to being able to generate images at varying resolutions, FOVs and orientation. Prior research has showcased that LDMs display state-of-the-art performance for generative modelling, and as such we will compare our results to that of an LDM identical in architecture to our proposed method without any spatial conditioning, trained solely on whole body CT images at a fixed resolution (192 × 192 × 208) . To compare to our geometry invariant model, we will take whole-body samples from the baseline LDM and apply transforms to generate the desired sample i.e. via scaling, cropping and rotating. We aim to showcase that our model showcases no performance degradation compared to a whole-body fixed resolution model. This is complimented with the added benefit of being able to control sample attributes without any reduction in sample quality. We test our model at 3 resolutions (96 × 96 × 104, 192 × 192 × 208 and 272 × 272 × 288) , in addition to 2 partial body views and a rotated orientation. To obtain quantitative metrics we use the Fréchet Inception Distance (FID) (Heusel et al., 2017) to measure how realistic the synthetic images are, with smaller values indicating images are more similar to the distribution of the real images. To calculate these values, we followed a similar approach to (Sun et al., 2022) , by extracting features using a pretrained Med3D (Chen et al., 2019) . For each experiment 1000 samples were generated to calculate the FID scores. Statistical tests were carried out via bootstrapping, and statistically significant findings (p < .01) are reported in bold in Table 1 . From table 1 we can generally see in cases where any scaling is required i.e. for the low and high resolution whole body experiments, our model shows superior performance, highlighting our model's ability to generate more realistic images and avoid any inaccuracies associated with resampling. With regards to the final three experiments that are simple transforms of rotating or cropping, we see roughly equal performance between both models, with our spatially conditioning LDM falling slightly short, albeit a statistically insignificant difference. It is not expected that our model would outperform in this case due to the nonintrusive transformations applied to the baseline, however this does showcase that under these conditions, our model generates consistent and structurally sound samples with almost no degradation in performance. 
experiment 2 -super-resolution: 
 Our second experiment showcases the emergent behaviour of our model for super-resolution without further training. For this we showcase low resolution samples and super-resolve them to 2 differing higher resolutions. For each sample we generate the super-resolved image 50 times and calculate the mean absolute error (MAE) between the original high resolution image and the super resolved images, in addition to the standard deviation across samples.We can see from Figure 4 that our model performs well to super-resolve the images to different resolutions, with limited error. Generally most of the error can be seen around the edges of large and higher intensity structures i.e. bone structures etc where finer detail is lost in the low resolution image. The standard deviation maps also highlight areas of high variation over each reconstruction. These maps are telling, as we can see from Figure 4 that there is a high correlation between the standard deviation and the mean error. This shows us that although there is some error, the model understands that it is uncertain around these areas showcased by the correlated high standard deviation and error.  
conclusion: 
 In our study, we have shown to effectively train a latent diffusion model to generative synthetic CT data at varying resolutions, fields of view and orientation. We showcase that our model is equal in performance to that of a model trained only on whole body data, with the necessary transforms applied post generation to produce a sample of given resolution or field of view and orientation. Even so in cases where scaling would be required, our model shows improved FID scores, showing greater consistency in the samples against inaccuracies brought about by resampling. Furthermore we show as an emergent behaviour of our training scheme, our model can be used to super-resolve images from any arbitrary lower resolution to any higher resolution, whilst also maintaining some awareness of areas of uncertainty and inaccuracies in the super-resolved images. This work brings to light the efficacy and potential of modelling not only the image, but the image geometry, represented by the affine to a group space with respective grid sizing to enable a better interaction with varying image characteristics. We hope this works warrants further efforts into such developments for a wider scope of applications in medical imaging. image. From equation 4 the first term seen is the L2 pixel loss, whilst the second term represents the spectral loss between the original and reconstruction. Here SFTF stands for the short-time Fourier transform. The third term is the commitment cost to ensure the encoder commits to the codebook. The final term is to move the codebook embedding vectors towards the encoder output. We replace this final term with an exponential moving average update for the codebook as implemented in (van den Oord et al., 2017) . During training, a β of 0.25 was used.To adjust the network to a VQ-GAN we additionally apply adversarial training with the existing VQ-VAE architecture and losses. In doing so the loss function is further amended to include a perceptual loss (Takaki et al., 2018) that helps to preserve spatial consistency, making use of the lpips library (Zhang et al., 2018) . This loss is randomly applied to 50% of slices across each plane.The architecture used for this work in the VQ-VAE makes use of an encoder with three downsampling layers containing convolutions with stride 2 and kernels of size 4, each followed by a ReLU activation and 3 residual blocks. Each residual block is made up of convolutions with kernel size 3, followed by a ReLU, another convolution with kernel size 1 and another ReLU. The decoder architecture mirrors that of the encoder except that it uses transposed convolutional layers with stride 2 and kernel size 4 for upscaling of the image. This work made use of a VQ-VAE codebook with 512 atomic elements (vocabulary size), each of length 8. An ADAM optimiser was used with a learning rate of 1e-4 and an exponentional learning rate decay with a gamma of 0.9999. Training was run for 600 epochs with a batch size of 3. 
 
 Appendix:  


Decision: The reviewers agreed that this paper presents an interesting approach (spatial conditioning) to enable latent diffusion models to be invariant to image resolution, orientation, and FOVs, with promising initial results demonstrated on whole-body CT images.

Reviews: review: paper_summary: paper_summary: The author present a spatial conditional method to take control over the resolution, field of view and orientation of image generated by the latent diffusion model. Moreover, the author showcase a emergent behaviour of super-resolution by repeated diffusion and denoising process.
main_review: main_review: The author clearly explain the intuition of the recurrent diffusion approach they applied for super-resolution and its bottleneck in VQ-VAE. Although the computing resource required for training is expensive which can potentially affect the reproducibility, I think this is a very interesting work that should be shared with the MIDL cohort.
strength_weakness: strength_weakness: 1. Similar to all the image synthesis study, the evaluation of the results is relatively weak. 

2.  The discussion about the super-resolution achieved by repeating the noising and denoising process seems to be insufficient. Is there any way to mathematically show why this is the case? Experimentally, is there an upper bound of the image resolution by taking more cycles of diffusion?
questions: questions: 1.  The discussion about the super-resolution achieved by repeating the noising and denoising process seems to be insufficient. Is there any way to mathematically show why this is the case? Experimentally, is there an upper bound of the image resolution by taking more cycles of diffusion? 

2. This approach requires a reference image representing the full spatial range [0, 1]. In this study, this standard is given by the whole body image. Is there any other modalities / data that can potentially benefit from this method? 

3. For the implementation details, how much memory will the latent diffusion model take for training? And how much time required to converge?
paper_summary: paper_summary: This paper presents a latent diffusion model that is invariant to the resolution and field of view of the data. The method consists of a VQ-GAN to obtain the sample latents and a DDPM based on these latents, both enriched with spatial encoding. The model is trained on whole-body CT images modified with different resolutions, FOVs, and orientation, demonstrating an improved generation quality as measured by FID in comparison to a baseline LDM without the spatial conditioning. An emergent super-resolution capability is also discussed and demonstrated.
main_review: main_review: The authors addressed my question on adding statistics to the results. My two other main comments were addressed as future work, although given the nature of the proof-of-concept I think the paper will be a good addition to MIDL.
strength_weakness: strength_weakness: The experiments considered only one dataset with a limited variety of FOV and orientations to demonstrate the real-world applicability of the method presented. 

The presented quantitative results miss statistics from multiple seeds of the model training results.

For image super resolution, it is not clear if it requires all the resolutions to be present in the training data?
questions: questions: The paper will be largely strengthened if more diverse settings of the FOV and orientations can be considered, or on datasets where FOV and orientations exhibit as more subtle changes of the images (rather then those shown in the whole-body CT scans).

Please add statistics of the quantitative numbers presented. 

Please address the question on whether the super-resolution is limited by the extend of resolutions presented in training images.
paper_summary: paper_summary: The authors use a diffusion model to generate more similar images with various properties (resolution, etc...) for improving the training of algorithms to be subsequently applied. They also show that this tool can naturally achieve the creation of superresolution images, which may also be of interest in this field.
main_review: main_review: Very good, I do not have any further comments.                                                                                                                                                            .
strength_weakness: strength_weakness: The only few weaknesses I would find are that a single dataset was used to assess the quality of the produced images, which may not be sufficient for a full-proof test of this algorithm, if one wants to use it on other types of images. There is also no example on how such images could be used later, and if they provide any added values.
questions: questions: I don't have any major questions to be addressed.

Comments: comment: Thanks for adding the standard deviation to the results. My other two questions were mostly responded as a future work, but seeing my fellow reviewers' comments, I'm happy to adjust my rating to a higher level.
Q1: The diagram in figure 1 could be improved a bit in terms of readability, some arrows/text are rendered very small, but there is space in the figure. 

A1: Thank you for your recommendation on the figure. We would of course want to ensure maximum readability and have amended figure 1 based off of your recommendations. 

 
Q2: I am not an expert on diffusion models, but this work seems of reasonable quality to be discussed. I would have liked a more thorough testing of this algorithm on other datasets, or comparison with previous methods, and an example of the added value of having produced more images. I would guess if they are too similar to the original dataset, the added value is limited, but if they are too different they will 'blur' the dataset and reduce the quality of later analysis. 

 
A2: We agree that further testing of this work on further datasets would have been useful in the validation, however the use of diffusion models in generative modelling on a range of modalities is common in the literature and has been shown to work, but we are equally sure that the suggested spatial conditioning would generalise to various modalities. As this paper focused on a new methodology, not validation we are of course limited by the page restrictions to showcase a wide evaluation whilst also communicating the method. However, we are pushing our future work to better evaluate these methods and showcase their utility on various datasets from varying modalities and dataset types. This would also include evaluation of further downstream tasks utilising the generated data. Nevertheless there are a number of works [1,2] that showcase the use of synthetic data to aid training for further downstream tasks that when complimented with real data improve performance, as such we are sure that this work can be beneficial to enhance the performance of other downstream tasks.

[1] - Fernandez, V., Pinaya, W.H.L., Borges, P., Tudosiu, P.D., Graham, M.S., Vercauteren, T. and Cardoso, M.J., 2022, September. Can segmentation models be trained with fully synthetically generated data?. In International Workshop on Simulation and Synthesis in Medical Imaging (pp. 79-90). Cham: Springer International Publishing.
[2] - Llugiqi, M. and Mayer, R., 2022, August. An empirical analysis of synthetic-data-based anomaly detection. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction (pp. 306-327). Cham: Springer International Publishing.
Q1: The paper will be largely strengthened if more diverse settings of the FOV and orientations can be considered, or on datasets where FOV and orientations exhibit as more subtle changes of the images (rather than those shown in the whole-body CT scans). 

A1: We agree with the reviewer that a more diverse set of FOV, orientations and further use in modalities would be beneficial in the evaluation of this work. Given this was a paper based on a new methodology as opposed to an extensive validation paper, we were limited in the length of the paper and the number of results that we can generate and report. 

That being said we can reassure the reviewer that given the nature of the training mechanism and use of diffusion models means the model is able to generalise and extrapolate to a variety of conditionings and diverse set of image geometries. Further work we have carried out has shown the ability of the model to generate a wider range of FOV’s i.e. smaller more concentrated regions of interest like just around the bladder and prostate or just the neck region. 

 

Q2: Please add statistics of the quantitative numbers presented. 

A2: We agree the importance of statistical measures in the quantitative numbers presented. The paper does highlight statistically significant results in bold in the table of results (p < 0.01) however reporting the standard deviation is also important and has been added to the results displayed in the paper. 

 

Q3: Please address the question on whether the super-resolution is limited by the extend of resolutions presented in training images. - it is not clear if it requires all the resolutions to be present in the training data 

A3: The resolution of samples is mainly limited by the resolution seen in the training data as a floor and a ceiling. This is also because the model relies on a VQ-VAE being able to reconstruct such images at the given dimension which is limited by the resolution in the training data. Nevertheless, diffusion models have strong generalisability properties at being able to extrapolate conditioning values [1]. Any intermediate values unseen in training data can still be generated within the lower and upper bound of what is seen in the training data. As the lower and upper bound of resolutions you would want to be reconstructed would be needed to train the VQ-VAE it is of course expected the same resolution ranges would be used to train the LDM, however an interesting follow up experiment would be to narrow the range of resolutions seen by the LDM (whilst the VQ-VAE is given the full range that it can reconstruct) and seeing if it can then extrapolate beyond the seen resolutions in training. We intend to explore this in future work. 

 

[1]- Pinaya, W.H., Tudosiu, P.D., Dafflon, J., Da Costa, P.F., Fernandez, V., Nachev, P., Ourselin, S. and Cardoso, M.J., 2022, September. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models (pp. 117-126). Cham: Springer Nature Switzerland.
Q1: The discussion about the super-resolution achieved by repeating the noising and denoising process seems to be insufficient. Is there any way to mathematically show why this is the case? Experimentally, is there an upper bound of the image resolution by taking more cycles of diffusion? 

A1: Regarding why the super-resolution approach works. It was first hypothesised we would be able to do so through the description of t-values and their influence on the structure of generated images in [1]. In this paper it was suggested that higher t values were responsible for larger structures and coarse features in images, with lower t values outlining the finer details. From this we hypothesised that this would equate to the general anatomical features of the body being defined early on in the denoising phase i.e. at high t-values, with finer details coming to light from the lower t values. Drawing parallels from super-resolution methodologies, generally these coarse features and structure is present in the low resolution data, and the finer features are those that need to be super-resolved. It was thus decided that running through the denoising steps at lower t-values for a low resolution image with a predefined structure would help to define these detailed features i.e. adding resolution to a low resolution image. Conversely using higher t-values would run the risk of altering the structure of the image. Further exploration into the relation between the latent representation of the same low and high resolution image would further showcase the structural similarity in the data with lacking details and highlight why this approach works. We agree further insight into this is useful and further research is being carried out to highlight the exact relationship and reasoning for this behaviour. 

As for the Upper bound of the image resolution, this is more so based on the VQ-VAE capabilities. We have found that super-resolved images via the LDM method can obtain comparable results to reconstructions of the VQ-VAE at the same resolution, however the output resolution of the VQ-VAE is limited by the data used during training, as the model would struggle to reconstruct data that is at a higher resolution than that seen during training. 

[1] - Ho, J., Jain, A. and Abbeel, P., 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, pp.6840-6851. 

 

Q2. This approach requires a reference image representing the full spatial range [0, 1]. In this study, this standard is given by the whole-body image. Is there any other modalities / data that can potentially benefit from this method? 

A2:The application of this study can be applied to various other modalities/data types. For example, PET can benefit from this method, especially looking at whole body PET, scans can generally have varying FOV – heart, head/neck and prostate, often at differing resolutions as well. Being able to model a particular modality at given FOVs and Resolutions can be beneficial to compliment such datasets. Furthermore, the arbitrary spatial range of [0, 1] can be representative of a different range. For example, if one was to solely work in neuroimaging one could use the [0, 1] range to represent a full head scan, where we would be able to model such scans at varying resolutions using this same methodology. 

Q3: For the implementation details, how much memory will the latent diffusion model take for training? And how much time required to converge? 

A3: For training the model was trained on 4 A100 GPUs with batch size of 6, for a total of 6000 epochs. The total training time for this was 5 days and 19hrs however it could be observed that good quality samples were present from ~ epoch 2500 with gradual improvements seen in quality over the remainder of training.
We would like to thank all the reviewers for their feedback. We are delighted to see that all reviewers find promise in the proposed spatial conditioning mechanism in addition to the model’s emergent behaviour used for super-resolution. There are however a number of common points brought to light by the reviewers that we would like to address. 

Firstly, regarding a more extensive evaluation and demonstration on further datasets showcasing varying modalities, fields of view and orientations. As this paper focused on the new methodology not validation, a large focus of the paper was the implementation of the spatial conditioning for generative modelling and its further use for super-resolution. As such we were of course limited by the page restrictions to showcase a wider evaluation whilst also providing an adequate overview of our methodologies. We agree with the reviewers that a more diverse set of validation sets ranging from different modalities to varying FOVs and orientations would be of interest to include. That being said diffusion models have already shown to be used in a range of medical imaging scenarios on various modalities and as such we have no doubt that this work would be applicable to a range of modalities. Additionally conditioning in Diffusion models has been shown to be very stable in prior works [1,2] and models are very capable of generalising and extrapolating to a variety of conditioning, which in this case would showcase images of a wide range of geometries. Research is currently being carried out to showcase more of these capabilities in a wider array of scenarios to better demonstrate and validate the abilities of this methodology. 

Additionally, regarding the super-resolution emergent behaviours and its limitations. Regarding the limit to the resolution achieved by this approach, generally it is seen that the main limitation to the resolution produced by the methodology is the floor/ceiling of the resolutions seen in the training data. Qualitatively it was seen that the super-resolution approach generates super-resolved samples of equal quality to that of what the VQ-VAE can reconstruct at the desired resolution. Naturally the limit of the resolution that the VQ-VAE can reconstruct is a result of the seen data during training. As such in a way the LDM itself is not limited in its performance for super-resolution and can perform on any data at any starting resolution to any end resolution provided that the VQ-VAE is able to accurately reconstruct that data. 

Although further small concerns have been noted by reviewers, no more comments have been recurring among the reviewers, and as such these have been addressed in the individual replies to each reviewer. 

We would once again like to thank the reviewers for their comments and for reading our paper. We look forward to addressing any further comments following the replies made. 


[1]- Pinaya, W.H., Tudosiu, P.D., Dafflon, J., Da Costa, P.F., Fernandez, V., Nachev, P., Ourselin, S. and Cardoso, M.J., 2022, September. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models (pp. 117-126). Cham: Springer Nature Switzerland. 

[2] - Rombach, R., Blattmann, A., Lorenz, D., Esser, P. and Ommer, B., 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).