# Memory-Based Learning and Fusion Attention for Few-Shot Food Image Generation Method
## 3 Method
### Image Generation Module  
In this section, an image generation module, including three attention mechanisms-Cross-modal Attention block (CmA), Memory Complementary Attention block (MCA), and Combinational Attention block (CoA)--is designed as shown in Figure 4. The image generation module, as illustrated in Algorithm 2, aims to refine the initial image features through the attention fusion module, aligning them as closely as possible with features in the attention region. These three attention mechanisms are defined as follows.  
Figure 4: Structure of attention fusion module.  
1. The CmA block is responsible for extracting interaction features between food ingredients and food images. Specifically, it integrates ingredient embeddings \(T_{i}\), \(Tm_{q}\), and image embeddings \(F_{en}\) to establish four distinct Cross-modal Attention blocks (denoted by \(CmA\_1\), \(CmA\_2\), \(CmA\_3\), \(CmA\_4\)), as shown in Figure 4. These equations of four CmA blocks are formulated as follows: \[CmA\_1=softmax\left(\frac{F_{en}\cdot T_{i}^{T}}{\sqrt{d_{k}}}\right)T_{i}\] (3) \[CmA\_2=softmax\left(\frac{F_{en}\cdot F_{en}^{T}}{\sqrt{d_{k}}}\right)T_{i}\] (4) \[CmA\_3=softmax\left(\frac{F_{en}\cdot Tm_{q}^{T}}{\sqrt{d_{k}}}\right)Tm_{q}\] (5) \[CmA\_4=softmax\left(\frac{F_{en}\cdot F_{en}^{T}}{\sqrt{d_{k}}}\right)Tm_{q}\] (6)
2. The intention of constructing the MCA is to adaptively learn the difference between image features retrieved from the Memory module. We assume that the difference between \(T_{i}\) and \(Tm_{q}\) indicates the similarity between the input ingredients and those stored in the Memory module. The formula for calculating this difference as the weight \(\alpha\) is defined as follows: \[\alpha=\left(\left|T_{i}-Tm_{q}\right|\right)\] (7)  
```
1 Input:\(Tm_{q}\), \(Im_{q}\), \(T_{i}\), \(D_{i}\), labels \(I\).
2 Output: Image \(I^{\prime}_{i}\).
3 for i = \(1,2,\cdots,B\)do
4 Fuse the image \(In_{q}\) and \(D_{i}\).
5 Input the fused image into the encoder of the U-Net, and obtain the encoder feature \(F_{en}\).
6 Input \(Tm_{q}\), \(T_{i}\), and \(F_{en}\) into the attention block.
7 Calculate Cross-Modal Attention \(CmA\_1\), \(CmA\_2\), \(CmA\_3\), \(CmA\_4\) by Formulas (3)-(6)
8 Calculate Memory Complementary Attention \(MCA\) by Formula (8).
9 Calculate Combinational Attention \(CoA\) by Formula (9).
10 Input the attention feature \(CoA\) into the decoder of the U-Net.
11 Output the image \(I^{\prime}\) generated by the decoder.
12 Calculate the loss between \(I^{\prime}\) and \(I\) by \(loss=\sum\limits_{i=1}^{B}{\left(I^{\prime}_{i}-I_{i}\right)^{2}/B}\).
13 Backpropagation and adjusting weight parameters of food image generation model.
```  
**Algorithm 2**Food image generation module.  
As shown in the equation above, a smaller difference implies a higher similarity; that is, the generated image will be more like the corresponding query image \(Im_{q}\). Therefore, the difference adaptively learned via the MCA attention Module to balance the contribution of image features and text features, defined as follows:  
\[MCA=\alpha\cdot F_{en}+(1-\alpha)\cdot T_{i} \tag{8}\]
3. The CoA block is designed to adaptively assign weight parameters \(\beta_{i}\) in both \(CmA\) and \(MCA\). For measuring the contributions of each attention, the fused attention feature \(CoA\) is formulated by a linear combination of these attention parameters, as shown Equation (9):  
\[CoA=\sum\limits_{i=1}^{4}{\left(CmA_{i}\times\beta_{i}\right)}+MCA\times\beta_{5} \tag{9}\]The procedure for the food image generation module is outlined in Algorithm 2.