Food image retrieval and generation
Creators: name: Han, Fangda
Publisher: name: No Publisher Supplied
Year: 2022

description: The domain of analysis and synthesis of food images is gaining increasing research interest due to its widespread applications in cooking and diet management. For instance, retrieval of food images from textual prompts can help speed up the cooking process. Likewise, extracting nutritional information from meal images can help monitor daily nutrient intake, facilitating diet management. The computational synthesis of photo-realistic food images is complementary to food image analysis. As an essential element of modeling the food image data, the synthesis also enables novel applications such as augmenting cooking instructions with multimedia content and gamiﬁcation of the food creation process to promote healthy eating habits amongst children. This dissertation focuses on developing computational tools for food image retrieval and generation. Our food image retrieval algorithm leverages the auxiliary information capturing relationships between related text-image pairs to regularize the latent space of food instructions and food images. Speciﬁcally, we develop a Coherence Aware Module (CAM) to augment the traditional text-to-image retrieval pipeline. The CAM is then trained to predict the auxiliary coherence relations that systematically characterize possible forms of relationship between related text-image pairs. Capturing these coherence relations has the effect of regularizing the learning of latent space embeddings of text-image pairs, resulting in accurate retrieval. Moreover, we show how CAM can be used to reﬁne queries during inference using the process of Selective Similarity Reﬁnement (SSR). Both CAM and SSR lead to signiﬁcant performance improvements in general text-image retrieval systems. Next, we develop a food image generation algorithm to generate images conditioned on multiple ingredients. First, we propose CookGAN, a novel extension of StackGANv2 with an explicit regularization in the form of Cycle Consistent Constraint (Cyc-constraint). Speciﬁcally, Cyc-constraint utilizes the pre-trained retrieval system discussed above to regularize the generation process and helps the model in generating images that more accurately reﬂect the desired content. However, CookGAN suffers from image blurring due to the limitation of model capacity. To address this problem, we propose Multi-ingredient Pizza Generator (MPG), an image synthesis approach that extends the StyleGAN2 architecture using a controllable conditioning input paradigm. Speciﬁcally, the control of ingredients relies on Scalewise Label Encoder (SLE) which helps the model to be strongly conditioned on the input ingredients while maintaining StyleGAN2’s excellent image quality. To verify the efﬁcacy of MPG, we validate it on Pizza10, which is a carefully annotated dataset of multi-ingredient pizza images. We show that MPG can successfully generate photo-realistic pizza images with the desired ingredients. However, while MPG can generate content-speciﬁc food images, it cannot control other image variation factors, such as the pizza shape, scale, or position, which are not available in the training data. To solve this problem, we propose Multi-attribute Pizza Generator (MPG2), together with Multi-Scale Multi-Attribute Encoder (MSMAE) and Attribute Regularizer (AR), targeting control of both ingredients and geometric attributes. We propose a cross-domain training schema to synthesize pizza images with the view attributes absent in the training dataset. This schema combines fully controllable computer graphics generated images (CGIs) with the partially annotated real-world data. To this end, we employ a view attribute regressor estimated on the CGI data to regularize the real-world food image generation process, thereby bridging the real-world and CGI training domains.