Hypernetworks for image recontextualization

Authors: maciej  zieba
jakub  balicki
tomasz  drozdz
konrad  karanowski
hong  lyu
aleksander  skorupa
tomasz  trzcinski
jakub m tomczak
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 17:26:12

Abstract: Image recontextualization, the task of placing a subject from an image into a new context to serve a specific purpose, has become increasingly important in fields like art, media, marketing, and e-commerce. Recent advancements in deep generative modeling, such as text-to-image and image-to-image synthesis via diffusion models, have significantly improved recontextualization capabilities. However, current methods, like DreamBooth and LoRA, require time-consuming fine-tuning per individual image, resulting in inefficiencies and often suboptimal outputs. Other approaches to recontextualization, like MagicClothing, require reorganization of the architecture of the base model and a time-consuming training process in a particular domain. In this work, we propose HyperLoRA, a novel framework that leverages hypernetworks to predict LoRA parameters, allowing for more efficient image recontextualization without the need for image-specific fine-tuning. HyperLoRA utilizes domain pairs of context images and target objects, enabling instant adaptation to new contexts while significantly reducing computational costs. Our method outperforms traditional techniques by offering more accurate adjustments, broader applicability across multiple modalities (e.g., text, video, sound, and structured data), and scalable deployment. Experimental results demonstrate the effectiveness of our approach in garment-to-model recontextualization, highlighting the potential for broader applications.

Main paper: 
 introduction: 
 Image recontextualization is a task that involves taking an image from its original context and presenting it in a new or different setting to serve a new purpose. This problem is typically encountered in art, media, and communication, and it is relevant to marketing and e-commerce. For instance, putting a photo of a purse captured by a seller in the context of a luxurious brand shopping window. In recent years, the concept of image recontextualization has gained more momentum due to deep generative modeling [1] , particularly image-to-image and text-to-image synthesis models like diffusion models. An example of such an approach is DreamBooth [2] which fine-tunes a large pre-trained text-to-image diffusion model to enable personalization and subject recontextualization. This model takes a few images of a subject and then generates new images contextualizing the subject in different environments, views, or poses while preserving key visual features.The problem in image recontextualization with pre-trained diffusion models is the necessity of finetuning. Fine-tuning of foundation models to specific tasks is a ubiquitous practice in machine learning, particularly in domains such as computer vision [3, 4, 5, 6] and natural language processing [7, 8] . Existing methods that tackle this problem, such as already mentioned DreamBooth [2] , or LoRA [7] , require fine-tuning per single image, which requires significant time and extensive prompting, while the results are often of low quality or do not match a given prompt.Recent techniques, such as IP Adapter [9] , enhance base architectures with additional, trainable cross-attention layers to preserve contextual details during recontextualization. However, these layers do not integrate context image information with textual features. Other methods, like MagicClothing [10] , necessitate the design of specialized architectures to address the problem.Here, we focus on improving fine-tuning of foundation models for image recontextualization. Unlike traditional methods that fine-tune models individually for each task or image, our approach leverages domain pairs comprising context images and target objects in desired contexts. For this purpose, we propose to utilize hypernetworks [11, 12] to predict LoRA [7] parameters, thus, we refer to our approach as HyperLora. The use of hypernetwork allows us to keep a single, shared neural network for outputting weights of a diffusion model. As a result, we eliminate the need for fine-tuning individual images, thereby significantly reducing computational costs and time requirements.Our approach offers several advantages over existing methods. Firstly, it enables instant adaptation to new context images without the need for fine-tuning, resulting in more efficient and scalable model deployment. Secondly, we demonstrate how to effectively combine features from both text and context image modalities without requiring additional architectural designs or training in the base model. Additionally, our method is not limited to image data but can be applied across multiple modalities, including text, videos, sound, and structured data (e.g., point clouds [13] ). Eventually, our hypernetwork provides a shared and unified representation for multiple contexts and data modalities.The contributions of the paper are as follows: (i) we propose a new framework for predicting LoRA parameters in a parallel manner for new contexts, (ii) we outline how to formulate and train hypernetworks for image recontextualization, (iii) we show how to effectively combine textual and context image features without modifying the base model architecture, (iv) we present experimental results demonstrating the effectiveness of our approach on garment-to-model image recontextualization. 
related work: 
 Fine-tuning Low-rank adaptation LoRA [7] is one of the most common fine-tuning techniques, initially adopted to Large Language Models. The idea behind this approach is to utilize two trainable low-rank matrices, that create the high-rank matrix after multiplication. This matrix is further added to the frozen matrices used in transformer layers and trained in a gradient-based manner. Thanks to the special initialization of the low-ranked matrices, the initial values of the matrix components are equal to 0, which implies that the values do not have an impact on the model at the beginning of the training. VeRA [8] is one of the possible extensions of LoRA, where thanks to specific decomposition, it is sufficient to train vectors, instead of low-rank matrices. Various extensions of VeRA are extensively studied in the work presented in [14] .Textual inversion [3] is a basic technique for fine-tuning diffusion models like Stable Diffusion [15] . The main idea behind this approach is to train the text embedding that represents a specific concept represented by a given image in a gradient-based procedure. Extended textual inversion [4] enriches the idea with multiple textual embeddings injected into different attention layers of the U-Net architecture. NeTI [5] is another method based on textual inversion that optimizes the mapping network instead of directly training the textual embeddings. One of the most common techniques for model fine-tuning is DreamBooth [2] . This method uses a unique textual token to represent the concept of the image and fine-tunes the entire U-Net architecture. In addition, the model uses specific regularization to preserve the generalization features. In a follow-up, [9] proposed HyperDreamBooth to adapt the idea of DreamBooth to fast personalization. This approach utilizes hypernetworks but differs from our method by a large margin. First, we formulate a parallelized hypernetwork, while the HyperDreamBooth uses transformer-based architecture. Second, our approach does not need a fine-tuning stage of LoRA parameters after training the hypernetwork. Finally, HyperDreamBooth is focused on personalization aspects, while our approach is more general and can utilize various modalities as context information.Recontextualization IP Adapter [6] is a multitask approach that can also be applied for recontextualization applications. The authors introduce effective and lightweight adapter to achieve image prompt capability for the pre-trained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Another approach, MagicFusion [16] , incorporates saliency-aware noise blending to preserve the details from contextual image during the fine-tuning approach. To preserve the details of the item, MagicClothing [10] uses two parallel UNets, one for processing garment features and the other for model generation. Compared to the described reference methods, our approach does not require problem-specific architectures and effectively combines textual and contextual information within a single cross-attention layer with no further fine-tuning. 
diffusion models: 
 Diffusion models Diffusion models [17, 18, 15] have risen to prominence within the generative model area, setting new baselines in the field of image generation [19, 20, 21] . Their utility extends to various practical applications such as enhancing image quality [22, 23] , compressing images [24] , image manipulation [25, 26] , colorization [27, 28] , style transfer [29, 30, 28] , among others.Diffusion models are a class of generative models designed to closely approximate the original data (images, or images conditioned on some additional information) distribution q(x 0 ) from image space I with model's distribution p θ (x 0 ):EQUATIONwhere x 1:T = x 1 , . . . , x T are variables of a forward Markov chain such thatx t = √ α t x 0 + √ 1 -α t ε, where α t is a so-called noise scheduler (a hyperparameter) such that 0 = α T < α T -1 < • • • < α 1 < α 0 = 1 assuring that x t converges to a Gaussian noise ε ∼ N (0, I) as t approaches T .Latent Diffusion Models Latent Diffusion Models (LDMs) [15, 31] improve the efficiency (and the quality) of diffusion models by operating in a latent space. LDMs use some (variational) autoencoder pre-trained on a large collection of images, with an encoder E that maps an image x into latents, z = E(x), and a decoder D such that D(E(x)) ≈ x. The diffusion model is then trained to produce representations of images in a latent space and could be seen as a marginal distribution over latents. For an image x 0 , the diffusion process adds noise to latent image z 0 = E(x 0 ) producing a noisy latent image z t . The amount of noise increases with t ∈ {1, . . . , T }. The input image x = x 0 can be conditioned on some additional information y, e.g., text prompt (text instruction), other image(s), etc. from space C. By D train we denote a training set o pairs (x, y) ∈ I × C. Let τ θ (y) be a model that maps conditioning input information y from space C into a vector of predefined dimension. The model's loss is defined as:EQUATION)where ε is a Guassian noise, t is a time step, z t is the latent noised up to time t, i.e., z t = √ α t + √ 1 -α t ε and ε θ is a denoising network. Roughly speaking, the objective is to correctly remove the noise which was added. Networks τ θ and ε θ are jointly optimized to minimize L LDM . At inference time, a random noise z T ∼ N (0, I) is sampled and iteratively denoised to produce latent image z 0 , which is afterward transformed into an image via decoder, i.e., x = D(z 0 ).Problem formulation In the case of a task for which only a few samples are available, e.g., generating images of a given style represented by a set of a few images, we could train a diffusion model from scratch. However, especially if the number of images is limited, fine-tuning a large diffusion models outperforms training the model from scratch. Roughly speaking, the procedure starts with a pre-trained model with parameters θ and gradually changes their values to θ ′ . The rationale is to adjust the model to the task at hand to improve its performance. This approach leverages the general capabilities learned during the initial training phase and adjusts the model's parameters mildly to better capture the nuances of the new data, thus, enhancing its ability to generate or process data.Fine-tuning as explained above is widely used to tailor generic models to specialized applications without the need for training a model from scratch. In this paper, we focus on a specific example, namely, we fine-tune a diffusion model for recontextualization using a collection of garments. Our goal is then to generate a realistic image of a human model wearing a specific garment keeping all the details (fabric, color, logos, etc.) of the garment. Formally, let us denote a number of new pairs (x, y) as D FT train , then the fine-tuning loss can be defined as follows:EQUATIONLosses L LDM and L FT are almost identical, the difference lies in the used empirical distribution and the initial weight values of the model. We assume that when optimizing the objective (3), we start with a pre-trained model, namely, θ resulting from (2), however, not all weights are optimized. For instance, DreamBooth [2] fine-tunes all parameters of the U-Net model ε θ , whereas Textual Inversion [3] (where the space C of conditional additional information is a set of prompts) fine-tunes a special token (not present in C) -its embedding to be more precise -in the CLIP text encoder.Attention-based methods An interesting line of research aims at modifying the text-to-image diffusion process through adjustments in cross-attention layers, allowing for more precise control over the images produced. In [32] , the capabilities of this approach are demonstrated in an example of editing actual images in various scenarios. More specifically, the cross-attention block adjusts the network's latent features based on conditional features, such as text features in text-to-image diffusion models. For given text features c ∈ R s×d and latent image features f ∈ R h×w×l , a single cross-attention operation consists ofQ = W q f , K = W k c, V = W v c.The operation then computes a weighted sum over the value features:EQUATIONwhere projection matrices W q , W k and W v map the inputs to a query, key and value feature respectively. Here d ′ is the output dimension of key and query features. The output of the attention layer is further transformed using the output linear projection layer parameterized by W o :EQUATIONThe aim of fine-tuning is to update mappings from a given prompt to image distribution, and text-based features appear only in W k and W v , that is why in [32] it is proposed to update only parameters of these two matrices.LoRA. Many recent approaches, like LoRA [7] , show the benefits of modifying all four W q , W k , W v , W o matrices instead of fine-tuning the parameters of W ∈ R d×s by updating their lower-dimensional representations (a product of two low rank matrices). LoRA keeps the pretrained weights frozen and injects trainable decomposition matrices (into each layer). Formally, let i = 1, . . . , L be the layer number. At layer i we have four pre-trained matrices W j i , j ∈ {q, k, v, o}. Let r ≪ min(d, s). The update is constrained by representing the latter with a low-rank decomposition, namely:EQUATIONwhere B j i ∈ R d×r , A j i ∈ R r×s . During training weights W j i are frozen, only A j i and B j i are trainable. For h j i = W j i x, the modified forward pass is the following:h j i = W j i x + ∆W j i x = W j i x + B j i A j i x.Moreover, B j i are initialized with zeros, i.e., A j i are initialized with random Gaussian noise, so that ∆W j i = 0 initially. In other words, the model kicks off from the original pre-trained matrices.  
hypermodels: 
 This work introduces HyperLoRA, a novel approach for recontextualization. Compared to reference approaches, like MagicClothing [10] , our method works like a plug-in model and does not require modifying the parameters of the base diffusion model. Our solution also does not require additional cross-attention mechanisms like the IP Adapter [6] , which injects the features of context image separately from the textual information. Instead, we utilize LoRA parameters to combine the textual and image context features together in the attention layers of the base diffusion model.In our approach, we postulate the use of hypernetworks, the networks that predict the parameters of the other model to adapt it to a particular context. The hypernetwork takes a context image as input and predicts the parameters that are usually fine-tuned using standard procedures like LoRA. Thanks to this approach, the set of parameters is adjusted to the new context image without needing a fine-tuning procedure, reducing time significantly.The rest of this subsection is organized as follows. First, we provide the architectural details of our approach. Second, we focus on how the model is trained using domain data. Finally, we describe how the trained model is utilized for recontextualization.The architecture of the proposed HyperLoRA model is provided in Figure 1 . The contextual image x c is fed to a hypernetwork H ϕ (•) parameterized by ϕ. H ϕ (•) is composed of a trainable CLIP image encoder and shallow parallelized hypernetwork layers that predict individual LoRA low-ranked matrices A j i , B j i for each of cross-attention matrices j ∈ {q, k, v, o} on each layer i ∈ {1, . . . , N } of the denoising network. During training, all W j i are frozen. The predicted matrices A j i , B j i are added to frozen W j i to the attention layers in the same manner as in LoRA-based fine-tuning, given by Eq. ( 6). The prompt for the target image is generated by merging the textual description of the context image with the description of the target context, e.g., attributes of a human model wearing a desired item given by the context image. For both image-to-text and merging operations, we use a pre-trained large language model, e.g., ChatGPT-4. The target image is generated using the input prompt in the same manner as the fine-tuned LoRA model.To train HyperLoRA, we use a paired set composed of context images describing a concept and its corresponding target image, together with conditioning textual description (prompt). The training set D train consists of triples (x c , x tg , y tg ), where x c is the context image, x tg is the corresponding target image and y tg is the target prompt.We apply training with respect to the parameters ϕ of H ϕ (•) by minimizing the following loss function:EQUATIONwhere ε θ ≡ ε θ (z tg t , t, τ θ (y tg ), H ϕ (x c )), and H ϕ (x c ) provides the LoRA parameters to the denoising model ε θ .Moreover, to emulate the zero-value condition for initialization of B j i matrices (B j i = 0), we incorporate additional, trainable scaling parameter γ (the same for all matrices, initially close to 0) and scale the matrices' values, i.e., in Eq. ( 6) the update is as follows:EQUATIONThus, the trainable parameters of the model are: CLIP image encoder, shallow hypernetworks (each predicting values of matrices A j i and B j i ) and γ ∈ R. The initial values of γ are set close to 0. With such initialization, the values of B j i do not affect the model performance at the early stage of the training.Assuming a trained hypernetwork H ϕ (•) and the context image x c together with semantic description of target context, y tgc , we aim at generating the target image x tg . The hypernetwork takes the context image x c , H ϕ (x c ), and predicts the LoRA parameters as it was described in Sect. 4.2. The semantic description extracted from context image x c and the description of target context, y tgc are merged together to create the prompt for the target image y tg . Finally, the target image is generated using the prompt y tr with the stable diffusion model with the LoRA weights predicted by hypernetwork H ϕ (•). 
experiments: 
 We evaluate our approach empirically and compare the results with selected reference methods adapted to contextualization tasks. First, we make the quantitative evaluation using metrics that are focused on maintaining consistency of generated and ground true images, comparing the results with reference approaches. Second, we focus on qualitative evaluation using both model variants with and without pose guidance.Dataset. We use the VITON HD [33] dataset that contains image pairs representing a top clothing image and a frontal-view human model wearing the corresponding item. The original dataset comprises 11, 647 training and 2, 032 testing pairs. Evaluation Metrics. Following the methodology from [2] , we use the following evaluation metrics, which are mainly focused on preserving consistency between ground truth and generated photos, namely:• Context to generated. (C2G). This metric represents the cosine similarity between the CLIP [34] embeddings of the context and the generated target images.• Target to generated. (T2G). This metric is calculated similarly as C2G, but a ground-truth target image is used instead of a context one.• Target to generated difference (T2Gdiff). This metric represents the absolute value of the difference between C2G and C2T, where C2T is the CLIP cosine similarity between ground-true context and target images. Methods. We compare our approach against selected reference methods, such as IP Adapter and MagicClothing. We selected those baselines because they utilize the entire training set to train the recontextualization method for a particular domain unlike standard fine-tuning methods (e.g., LoRA or Dreambooth) that utilize few images to fine-tune for a single item.We adopt a two-stage training approach for our method. Initially, the model is trained using a static prompt, 'A woman wearing S*'. In the subsequent stage, the target prompt is generated by merging the descriptions of the conditioning image and the target context (attributes of the model wearing the item, see Figure 1 ). For tasks involving image-to-text conversion and text merging, we leverage ChatGPT-4. Two evaluation variants are considered for each model used in the experiments. The first, referred to as the standard approach, generates the target image based solely on the context image.The second variant incorporates pose guidance, generating a target image of the model in a specified pose while wearing the item from the context image. This is achieved by utilizing model weights integrated into the ControlNet [35] architecture. Additionally, we evaluate a variant of our model combined with IP Adapter to improve consistency between the generated and target imagesWe employed the VITON HD test set [33] , which consists of 2,032 images, for quantitative evaluation. For these images, we calculated the C2G, T2G (where higher values indicate better performance), and T2Gdiff (where lower values indicate better performance) metrics.Table 1 presents a comparison of the computed metric values across all generated images. The results show that the HyperLoRA variants consistently outperform the baseline methods across all metrics, demonstrating superior performance. When utilizing poses aligned with the target images, we observe a slight reduction in the T2Gdiff score and a marginal improvement in the T2G metric. Additionally, the integration of the IP Adapter with the HyperLoRA model leads to an increase in consistency between the context and generated images, as reflected by the improved C2G metric.In Tables 2 and 3 , we provide qualitative results comparing the performance of different methods, both without and with pose guidance. The methods based on HyperLoRA demonstrate superior preservation of contextual information compared to other techniques, particularly in simpler cases, and show mostly improved performance in moderately challenging cases. Additionally, the poses generated by our proposed methods exhibit a more natural appearance. Notably, the combination of HyperLoRA with the IP Adapter appears to offer enhanced preservation of item details. However, it is important to acknowledge that all methods evaluated exhibit some degree of inconsistency in preserving the details of certain logos across the experiments. This phenomenon is observed due to the fact that HyperLoRA compresses the input image into a small embedding vector, which may lead to losing important high-frequency details. Therefore, in future works, we will investigate incorporating the parallel Unet concept provided in MagicClothing into our approach, in order to preserve both high-level and low-level features.  
conclusions: 
 The research introduces a novel method for instant recontextualization in fine-tuning foundation models, addressing shortcomings of existing approaches such as DreamBooth and LoRA. Unlike traditional methods, which require fine-tuning per single image and extensive prompting, the proposed approach leverages domain pairs and hypernetworks to predict parameters, eliminating the need for fine-tuning individual images and significantly reducing computational costs and time requirements. The method offers several advantages, including instant adaptation to new context images without fine-tuning, more accurate adjustments, and applicability across multiple modalities. Compared to methods that utilize context images like IP Adapter or MagicClothing our approach combines textual and image features in one attention mechanism and does not require any modifications and training of the base generative model. Experimental results demonstrate its effectiveness, particularly in garment-to-model recontextualization, highlighting its contributions to the field.For future work, we would like to extend our method using a variety of modalities, including text, metadata, video, and speech. The adaptation of our model can be achieved simply by replacing the trainable image CLIP encoder with another type of encoding network suitable for such data. We also plan to extend the applications of HyperLoRA to other categories, considering the target and contested images in more sophisticated concepts.The other dimension, where our method can be extended, touches the representation of the predicted parameters for our model. In our studies, we focus on predicting the values of matrices of low-rank matrices used in LoRA-based decomposition. However, our framework can be successively applied to other decomposition methods, like VeRA [8] or other approaches described in [14] . 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Promising and Novel Approach for Image Recontextualization with Room for Further Comparison
main_review: main_review: ## Summary

The paper proposes a novel method for image recontextualization, specifically focusing on the task of placing an object or subject from an image into a new, specified context. The method introduces a **hypernetwork** that generates parameters for **LoRAs** (Low-Rank Adaptations), which are then incorporated into a generative model (Stable Diffusion). Conditioned on a given input image, the hypernetwork predicts the necessary LoRA parameters, which, along with a textual description of the new context, guide the model to produce an image of the input subject within the new setting. The paper presents promising results on a garment-to-model recontextualization task and suggests potential applications in various domains beyond the demonstrated use case.

## Strengths and Weaknesses

Strengths:

- The paper is well-written and easy to follow.
- The method is novel, it is simple and it works. It is easy to imagine this method being applied to other modalities.

Weaknesses:

- In the **related work** section, there is limited discussion of methods that enable real-time conditioning based on a single image, such as **MagicClothing** and **IP-Adapter**. These approaches are relevant because they avoid the need for image-specific fine-tuning, similar to the proposed method. A more thorough comparison of these real-time techniques would strengthen the paper, especially in terms of how HyperLoRA differentiates itself from these methods.
- In Table 2 and 3, it seems that HyperLoRA makes some visible mistakes when recontextualizing the black t-shirt. On the other hand, MagicClothing seems to get it right. This may be due to the way HyperLoRA’s image encoder compresses the input image into a small embedding vector, potentially losing important high-frequency details. By contrast, MagicClothing’s U-Net architecture likely handles such details more effectively. A more detailed analysis of these errors and a comparison between the methods would add value to the paper.

## Questions/Suggestions

- The sentence in line 85 seems to contain an error: “[…] is a multitask approach **that solves** that also”. Consider revising this sentence to remove the redundancy.

Soundness: 3

Presentation: 3

Contribution: 3
title: title: Accept - Hypernetworks for image recontextualization
main_review: main_review: The paper proposes HyperLoRA, a novel approach leveraging hypernetworks for image recontextualization without the need for image-specific fine-tuning. One of its key strengths lies in its ability to reduce computational costs and time compared to existing methods like DreamBooth and LoRA, which require extensive fine-tuning for each image. Another strength is the model's adaptability across multiple modalities (e.g., text, video, sound), making it versatile for various recontextualization tasks. The experimental results demonstrate its superior performance in garment-to-model recontextualization, and the qualitative comparisons further showcase its effectiveness in preserving item details and generating natural-looking results, particularly when combined with pose guidance.

However, potential weaknesses include the limited exploration of applications beyond garment-to-model recontextualization. Additionally, while the paper highlights performance improvements over IP Adapter and MagicClothing, more detailed comparisons with additional state-of-the-art methods could provide a more comprehensive evaluation. Expanding experiments to include more complex, real-world scenarios and larger datasets would strengthen the argument for broader applicability. Overall, HyperLoRA shows great promise but could benefit from further validation in diverse contexts.
title: title: HyperLoRA = Hypernetworks for LoRA
main_review: main_review: This paper proposes to predict LoRA parameters using a hypernetwork. This allows to improve contextualization in the generated images. Second, HyperLoRA offers a more efficient and scalable deployment because it enables instant adaptation to a particular context. Experiments show advantages of incorporating image and textual contexts to image generation using the stable diffusion model. Overall, HyperLoRA might be a practical method for recontextualization even though it has an incremental academia contribution.