# Inv-ReVersion: Enhanced Relation Inversion Based on Text-to-Image Diffusion Models
## 3 Method
### POS Weighted Control Loss  
ReVersion [12] introduced \"preposition prior\", noting that prepositions can sparsely activate relations between entities in images. To guide the relation vector \(\langle R\rangle\) into the preposition space of text embedding, ReVersion designed a relation-steering contrastive loss based on InfoNCE contrastive learning:  
\[L_{\text{ReVersion}}=-\log\frac{\sum_{l=1}^{L}e^{R^{T}\cdot P_{i}^{l}/\gamma}}{ \sum_{l=1}^{L}e^{R^{T}\cdot P_{i}^{l}/\gamma}+\sum_{m=1}^{M}e^{R^{T}\cdot N_{m }^{l}/\gamma}} \tag{3}\]  
where R is the relation vector; \(P_{i}=\left\{P_{i}^{1},\ldots,P_{i}^{L}\right\}\) (i.e., positive samples) refers to the preposition vector randomly sampled in the ith iteration; \(N_{i}=\left\{N_{i}^{1},\ldots,N_{i}^{M}\right\}\) (i.e., negative samples) refers to word vectors of other POS randomly sampled from the word embedding space.  
However, prepositions are mainly used to activate spatial relations in space, and they perform poorly in activating behavioral relations (such as \"carved\", \"made of\"). In contrast, verbs are often more intuitive and accurate in representing behavioral relations [46; 47]. Therefore, we propose to increase the influence of verbs in contrastive loss to more effectively reinforce the embedding of relation vectors similar to behavioral relations.  
Figure 3: In the ReVersion method, as shown in the first row of the figure, when the dog appears before the cat in the text prompt, the cat on the left exhibits dog features (e.g., a longer mouth). Conversely, as shown in the second row, when the cat appears before the dog in the text prompt, the dog exhibits the patterns of the cat on the right. However, the model trained with inverse relation text expansion can effectively solve this feature fusion problem between multiple entities.  
Consequently, our proposed POS weighted contrastive loss is as follows:  
\[L_{control}=-\big{(}\lambda_{verb}L_{verb}+\lambda_{prep}L_{prep}\big{)} \tag{4}\]  
\[L_{verb}=\log\frac{\sum_{l=1}^{L}e^{R^{T}.P_{Vi}^{l}/\gamma}}{\sum_{l=1}^{L}e^{ R^{T}.P_{Vi}^{l}/\gamma}+\sum_{m=1}^{M}e^{R^{T}.N_{i}^{m}/\gamma}} \tag{5}\]  
\[L_{prep}=\log\frac{\sum_{l=1}^{L}e^{R^{T}.P_{Vi}^{l}/\gamma}}{\sum_{l=1}^{L}e^{ R^{T}.P_{Vi}^{l}/\gamma}+\sum_{m=1}^{M}e^{R^{T}.N_{i}^{m}/\gamma}} \tag{6}\]  
\[\lambda_{verb}+\lambda_{prep}=1 \tag{7}\]  
where \(L_{verb}\) represents the verb-steer contrastive loss, \(L_{prep}\) represents the preposition-steer contrastive loss, \(\lambda_{verb}\) represents the weight of the verb loss, \(\lambda_{prep}\) represents the weight of the preposition loss, \(P_{Vi}=\big{\{}P_{Vi}^{1},\dots,P_{Vi}^{L}\big{\}}\) represents the verb samples, and \(P_{Pi}=\big{\{}P_{Pi}^{1},\dots,P_{Pi}^{L}\big{\}}\) represents the preposition positive samples.  
The advantage of this method is that, compared to directly replacing positive samples from prepositions with verbs, our weighting strategy can flexibly adjust the tendency of the relation vector towards spatial or behavioral relations according to the type of relation. In this way, this method will perform better in learning behavioral relation vectors than using only prepositional relations as positive samples.