LEAF: Predicting the Environmental Impact of Food Products based on their Name (2024)

Abstract: Although food consumption represents a substantial global source of greenhouse gas emissions, assessing the environmental impact of off-the-shelf products remains challenging. Currently, this information is often unavailable, hindering informed consumer decisions when grocery shopping. The present work introduces a new set of models called LEAF, which stands for Linguistic Environmental Analysis of Food Products. LEAF models predict the life-cycle environmental impact of food products based on their name. It is shown that LEAF models can accurately predict the environmental impact based on just the product name in a multilingual setting, greatly outperforming zero-shot classification methods. Models of varying sizes and capabilities are released, along with the code and dataset to fully reproduce the study.

text: Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024), pages 133–142 August 16, 2024 ©2024 Association for Computational Linguistics
Although food consumption represents a substantial global source of greenhouse gas emissions, assessing the environmental impact of off-the-shelf products remains challenging. Currently, this information is often unavailable, hindering informed consumer decisions when grocery shopping. The present work introduces a new set of models called LEAF, which stands for Linguistic Environmental Analysis of Food Products. LEAF models predict the life-cycle environmental impact of food products based on their name. It is shown that LEAF models can accurately predict the environmental impact based on just the product name in a multilingual setting, greatly outperforming zero-shot classification methods. Models of varying sizes and capabilities are released, along with the code and dataset to fully reproduce the study.

heading: 1 Introduction
text: Reducing global greenhouse gas emissions is a key objective for mitigating rapid climate change. Recent estimates based on life-cycle assessment (LCA) data say the global food system accounts for up to 30% of global greenhouse gas emissions (Li et al., 2022). Although one can not completely eliminate the emissions from food consumption, the environmental impact can be greatly reduced by avoiding foods with a high climate impact. For most food products however, this information is not readily available, which makes it more difficult for consumers to make informed decisions1.
In this work, a new set of models is introduced which can predict the environmental impact of a product based on the product name. These models learn relationships from products with existing LCA data, which can subsequently be applied to any text.

heading: 1.1 Related Work
text: In recent years a handful of studies have described models that predict certain aspects of food products. Hu et al. (2023) use a BERT model for food classification of Canadian branded products in the context of nutrition. Balaji et al. (2023) do emissions estimation of general consumer products using zero-shot classification based on a sentence BERT model.
The Open Food Facts (OFF) data (Section 2.1) is an excellent resource for the current work. OFF has developed a computer vision model called Robotoff (2024(b)) which predicts missing data fields like category, weight and brand based on the product image. These predictions are subsequently verified in a crowd-sourcing environment called Hunger Games (2024(a)).
To the best of available knowledge, the current study is the first work exploring the usage of NLP methods specifically for the estimation of environmental impact of food products.
1On Open Food Facts, 73% of products have no Eco-Score
133

heading: 2 Methods

heading: 2.1 Open Food Facts Dataset
text: The OFF database consists of open-access, crowdsourced products. With 40% of products, the dominant language of the database is French, followed by English (32%) and Spanish (10%). Although the OFF platform is mostly popular in France, the website has dedicated pages for a large variety of countries and territories, and contributions to the database are made by consumers and producers globally. The platform has quality control measures like community review and error detection when uploading a product. The product entries vary in completeness, where some products have a detailed list of ingredients and others just a picture and a name. A more elaborate data analysis can be found in Appendix A.
The dataset is filtered for products that have an associated Agribalyse class (Colomb et al., 2015). In total there are 2518 Agribalyse classes present in the dataset, each of which has an associated life-cycle assessment (LCA) that estimates the environmental impact measured as the environmental footprint (EF) score (Colomb et al., 2015; European Commission, 2021). The EF score is a weighted combination of 14 different factors, expressed in millipoint (mPt) per kilogram of product. EF score factors are related to the full lifecycle of a product, including manufacturing, packaging, transport, consumption and disposal. The biggest contributing factor in this score is the climate impact, measured in carbon dioxide equivalent or CO2Eq (Brander and Davis, 2012). This unit is also used to compute the Eco-Score (Facts, 2023) as displayed on the OFF website (Figure 1). While the concept of a discrete A/B/C/D/E rating system for the Eco-Score is similar to that of the widely-adopted Nutri-Score (Chantal et al., 2017), the formulae and methodology behind the two label values differ.
The OFF database primarily gathers its data
through crowdsourcing. The dataset is licensed under the Open Database License (ODBL) (Open Data Commons). The dataset used for this work was exported on March 31st 2024.

heading: 2.2 Task and Models
text: The current work introduces a set of models to predict the EF score of a product based on its name. A high-level task overview is given in Figure 2.
LEAF models consist of a pretrained sentence embedding base model and a readout head. The distiluse-multilingual-base-v2 (DU) transformer model (135M parameters) (Reimers and Gurevych, 2019) and a larger bge-m3 (M3) transformer model (561M parameters) (Chen et al., 2024) are used due to their cross-lingual capabilities where semantically similar texts across languages are nearby in vector space. The parameters of the base model are frozen and not fine-tuned; instead the static sentence embeddings serve as input for the learnable task-specific heads. Three different LEAF model configurations are introduced, based on their unique readout heads:
• LEAFc: Standard classification head comprising a dense layer with 2518 output nodes, followed by a softmax function and optimised by a cross-entropy loss function.
• LEAFr: Regression head that predicts a single continuous value using a dense layer, followed by a softplus function that maps the output to a non-zero positive value and optimised by a MSE loss function.
• LEAFh: Hybrid head that combines the classification head and the regression head in a sequential way, such that each logit can contribute individually to the resultant regression value. Both the logits and the regression output are simultaneously optimized, with a hyperparameter α (set to 0.5 in experiments) controlling the weight of the individual loss terms (details in Appendix B.1).
The different model configurations are illustrated in Figure 3.

heading: 2.3 Experimental Setup
text: Models are evaluated based on the predicted deviance from the ground truth EF score, as measured by the mean absolute error (MAE). For the classification model, the EF score is computed by mapping the predicted class to the corresponding value. For the other model heads, the predicted EF score is the actual regressand.
Models are tested on a holdout set comprising 20% of the original training data. The test set is randomly sampled using stratified sampling based on the product language, such that the language distribution of the test set reflects that of the training set. Metrics are micro-averaged across all data points. For each LEAF configuration, a grid search is performed to identify good values for the batch size and learning rate (parameters in Table 5). The grid searches use the smaller DU embedding model, and the best model configuration is also trained using the M3 embedding model.

heading: 2.4 Baseline Models
text: LEAF models are evaluated against two different types of baseline models. The first baseline model is a zero-shot autoregressive classifier using the OpenAI gpt-3.5-turbo model API (175B parameters) (Brown et al., 2020). For budgetary reasons, this model is chosen over the more powerful gpt-4-turbo variant and the test sample is deliberately smaller to reduce API costs. The model achieves an accuracy of 0.374 and a MAE of 0.110
on a random sample of 1000 products when ignoring any hallucinated class predictions. When considering hallucinations as random guesses, the accuracy is corrected to 0.302. Optimising and paraphrasing the textual prompt has negligible impact on performance. Details of the exact baseline methodology are supplied in Appendix B.2.
In addition, an untrained DU and M3 are evaluated on the entire test dataset. Predictions are obtained by constructing an embedding table by embedding each of the class names. Given an embedded product name, its predicted class is defined by taking the class with the lowest cosine similarity. Although the M3 model with CLS pooling achieves the best open-source baseline performance with an accuracy of 0.193 and a MAE of 0.300, it does not outperform the OpenAI baseline model. Interestingly, the mean-pooled DU embedding models seem to substantially outperform CLS-pooled ones.

heading: 3 Results
text: The accuracy and MAE values of different models are summarised in Table 1. There is a clear performance gap between classification and regression models, where classifying products in concrete classes seems to result in higher accuracies and lower MAE scores than predicting a continuous value. Among classification models, LEAFc outperforms the OpenAI baseline despite having about a thousand times less parameters. Although the accuracy of the OpenAI model is substantially lower than that of LEAFh, the MAE of OpenAI is actually better, which implies that misclassifications of the OpenAI model are typically within a smaller error bound than those of the more accurate LEAFh model. A sample-based qualitative comparison of the OpenAI baseline and LEAFc models supports this claim (details in Appendix C.1). Note that the MAE score of 0.071 is considerably smaller than the global dataset standard deviation of 0.448, and that a more granular class distribution like the Eco-Score is less sensitive to small numerical deviations.
Ablation studies are performed on the bestperforming LEAFc model configuration to gain deeper insights into which configurations contribute to it's performance. The results are summarized in Table 2. Firstly, it is noteworthy that the M3 model brings an increased performance versus the DU model in a classification setting. This can be explained partly by the larger parameter count
of the base model (561M parameters versus 135M). Secondly, using CLS pooling, we observe a slight drop in performance compared to the mean-pooled configuration, which is also as expected considering the baseline scores for the DU models. Lastly, finetuning the last attention layer of the DU model while training the classifier results in a sharp performance decrease. This can have several causes, but it seems likely that the model starts overfitting the attention mechanism to the task at hand, losing meaningful capabilities attained during pretraining (Ramasesh et al., 2021).
An analysis of multilingual performance shows that LEAFc, M3 performs best for the top-5 languages in the dataset. The results can be found in Tables 9 and 10 of Appendix C.2.

heading: 4 Conclusion
text: Modern NLP methods can accurately predict the environmental impact of food products using only their names across various languages. Empirical evidence shows that classification is preferred over regression for estimating EF scores. Among different model configurations, LEAFc models substantially outperform others in both accuracy and error, and they also surpass GPT-3.5 in a zero-shot classification setting. All in all, predictions based on the product name are a simple yet powerful approach, and LEAF models can be considered for tasks like Eco-Score prediction.

heading: 5 Limitations and Future Work
text: While LEAF offers a novel approach to predict EF scores for a variety of products, this work has
certain limitations which are transparently outlined to raise awareness and encourage further research.
Limited Class Specificity: There are no individual differences within an Agribalyse class. For example, an apple belongs to the apple class, regardless of whether that apple is produced locally or overseas. Various factors influencing a product's environmental impact are abstracted and averaged out, although the difference can be significant. Future work can address this by e.g. using more fine-grained LCA data or by working on EF score explainability.
Fixed Consumption Location: Current models assume the product is consumed in France, as per Agribalyse assumptions. The effects of certain large emission factors, such as transportation, are location-specific and substantially contribute to greenhouse gas emissions (Li et al., 2022). Therefore, caution is needed when interpreting LEAF results for locations with significantly different food supply chains than France.
Additional Data Sources. The current work examines the relationship between product name and environmental impact. Other (potentially unlabelled) data sources, such as ingredient lists, country of production, country of consumption, transportation method, and packaging data, could provide additional insights for more accurate environmental impact predictions. A new model that combines different data sources under varying levels of uncertainty could be superior.
Processing of LCA Data. The current dataset has redundancy among certain classes. For example, there are three classes for almonds (peeled, unpeeled, and salted), all with the same LCA values. A compressed mapping for a new class distribution specific to EF score estimation could improve stability and performance by reducing the parameter count.

heading: A Open Food Facts Data Analysis
text: The OFF dataset has a skewed class distribution and a skewed language distribution with French being its most prevalent language.
In total, there are 800,589 products in the dataset. Products have an average EF score of 0.448 (σ = 0.454) and percentile values of 0.175, 0.310 and 0.588 for the 25th, 50th and 75th percentiles respectively. The 5 most-common classes are summarised in Table 3. The 5 products with the highest EF score all relate to lamb meat prepared in different ways, with a EF score of 5.813. The top-1 percentile of highest EF score products (n = 25) have an average EF score of 5.243 (σ = 0.594) and are composed of different types of meat (n = 21), different types of seafood (n = 3) and decaf instant coffee (n = 1).

heading: B Supplemental Information on Methods

heading: B.1 Hybrid Loss Function
text: In total, three different loss functions are implemented to train the LEAF models, of which one is a custom implementation. The cross-entropy loss and mean-squared error loss are common loss functions for training classification and regression models respectively. The hybrid loss function is defined as:
Lh = α ∗ LCE + (1− α) ∗ LMSE (1)
Where α is a non-negative value between 0 and 1. Although α could be a learnable parameter, this could lead the model to finding a local optimum by learning a value close to either 0 or 1 and thereby eliminating one of the loss terms. All experiments in the current work have a constant α value of 0.5.

heading: B.2 OpenAI Baseline Methodology
text: The OpenAI model was evaluated on a limited set of 1236 samples due to financial constraints. Of these, the model was able to make a valid prediction for 1000 samples, indicating a hallucination rate of 0.191. For these samples, the model had an accuracy of 0.374. When considering the invalid predictions as random guesses on a balanced dataset, the model achieves an accuracy of 0.302. The context length of this model is 16k tokens, which is insufficient to encode a single sample in one prompt including the model instructions, product name and all possible classes. To accommodate for the context window limit, the total number of classes are split into two random partitions for each test sample. The model in total does 3 classifications per sample; one for each of the two partitions, and another one to choose between the two partition classifications. The categories and partition splits are randomly shuffled for each sample. The generation temperature is set to 0 and the seed is set to 42. API calls were made on April 28th 2024 using the OpenAI Python SDK.
The system prompt consists of the following text:
You are a helpful assistant. Your task is to classify the text string given by the user. The string can be presented in any language. You must pick a class from the permitted categories you are provided, even if the correct class is not in the list.

heading: B.2.1 Handling Hallucinations
text: A significant challenge of the baseline model is its tendency to hallucinate new class labels that would fit the sample. If the model hallucinates a nonexistent category in one of the partitions, the other category is automatically picked as the predicted class. If both partition categories are hallucinated, the prediction is rendered invalid. Baseline metrics are provided for two scenarios: one where invalid predictions are considered random guesses and one where they are not considered. The probability that the model hallucinates is likely higher for more difficult samples, so it is important to interpret both numbers. Hallucinations can be prevented by limiting token generation to the possible class names, but to the best of available knowledge this is not currently supported in the OpenAI API.

heading: B.2.2 System Prompt Sensitivity
text: Since it has been shown that prompting can have a significant impact on model performance (Mizrahi et al., 2023; White et al., 2023), an additional experiment is performed to establish the sensitivity of gpt-3.5-turbo to system prompt variability for the classification task. Apart from the forementioned original system prompt, the following system prompts were evaluated:
1. Paraphrased: You are an assistant dedicated to providing support. Your objective is to categorize the text provided by the user. This text may be in any language. You must choose a category from the allowed list of options, even if the most appropriate category isn't included.
2. Minimal: Your task is to classify the text string given by the user. The string can be presented in any language. You must pick the correct class from the list of permitted categories, even if the correct class is not in the list.
3. Linguist: You are an expert linguist and text classifier. Your task is to classify the text string given by the user. The string can be presented in any language. You must pick the correct class from the list of permitted categories, even if the correct class is not in the list.
4. Environmentalist: You are an expert in assessing the environmental impact of food products. Your task is to classify the text string given by the user. The string can be presented in any language. You must pick the correct class from the list of permitted categories, even if the correct class is not in the list.
The methodology mostly is unchanged except for a reduced sample size (200 valid predictions
per prompt instead of 1000) and the experiment time (API calls are made on June 24th 2024 instead of April 28th). The results are summarised in Table 4. Firstly, the original assistant prompt has a lower accuracy of 0.345 (n = 200) than in the earlier experiment where it had an accuracy of 0.375 (n = 1000). This can be a result of intermittent OpenAI model updates that made the model slightly worse on the task. It can also be due to difference in sample size. Furthermore, it can be observed that the paraphrased prompt has a slightly higher accuracy than other variants, and that the original prompt has a slightly lower MAE and hallucination rate. It seems sensible that hallucinations are more likely for ambiguous (and therefore harder) samples, making it hard to objectively assess the efficacy of individual prompts. Nevertheless, prompt variation empirically has a negligible effect on overall performance, where LEAF models yield a substantially better performance than zero-shot classification models based on GPT-3.5. The accuracy can be further improved by utilising a more powerful language model or formulating as a few-shot classification task.

heading: B.3 Training Hyperparameters
text: The grid search hyperparameters are summarised in Table 5. Only the readout head, peak learning rate and batch size are varied among models. The random seed is set to a constant value of 42. For LEAFc, LEAFr and LEAFh, the best observed peak learning rate values are 256, 64 and 256 respectively. Similarly, the best observed batch sizes are 5e-2, 5e-3 and 5e-3 respectively.
For M3 base models, it is observed that a lower learning rate is required than for LEAFc using a DU base model. Based on 3 training runs with learning rates of 5e-2, 8e-3 and 5e-3, it is observed that 5e-3 performs best. For the LLFT ablation run, a classifier learning rate of 5e-2 and a attention layer learning rate of 5e-3 were used.

heading: B.4 Reproducibility
text: The codebase is available on GitHub2, including scripts for dataset creation and model training. The trained LEAFc with DU and M3 base models are available on the Hugging Face model hub under the aliases baskra/leaf-base and baskra/leaf-large respectively. The train and test datasets are available on the Hugging Face datasets hub under the alias baskra/leaf.

heading: C Additional Results

heading: C.1 Qualitative Analysis
text: A random sample of test set predictions is analysed to compare qualitative differences between the OpenAI baseline and LEAFc with DU and M3 base models, across each of the major languages in the dataset.
Table 6 shows misclassifications from DU where GPT-3.5 predicts the correct class. It is observed that GPT-3.5 misclassifications are generally more precise than the misclassifications of DU, which are relatively coarse. For example, GPT-3.5 correctly classifies the product name “Kräuteressig” as Vinegar, whereas DU misclassifies it as Camembert. This is notably in line with the observed metrics: while GPT-3.5 has a worse overall accuracy, it achieves a better MAE compared to DU. Conversely, Table 7 shows misclassifications of GPT-3.5 where DU predicts the correct class. Here is seen that GPT-3.5 misclassifies the product name “Chipolata aux herbes au sel de l'ile de re” as Sausage, which, while correct, is not as specific as the ground-truth Chipolata class.
Lastly, a comparison between LEAFc variants is made in Table 8, which shows misclassifications of DU where M3 predicts the correct class. Here one can see that both coarse and precise misclassifications of DU are not made by M3, indicating that the larger M3 model is overall more accurate than DU.
2URL: https://github.com/baskrahmer/LEAF

heading: C.2 Multilingual Performance
text: Metrics for the most-frequent languages in the dataset are present in Table 9. In addition, metrics for the most-spoken languages (according to (SIL International, 2023)) are present in Table 10.

heading: Future Work
text: Future work can address this by e. g. using more fine-grained LCA data or by working on EF score explainability. Fixed Consumption Location: Current models assume the product is consumed in France, as per Agribalyse assumptions. The effects of certain large emission factors, such as transportation, are location-specific and substantially contribute to greenhouse gas emissions (Li et al. , 2022). Therefore, caution is needed when interpreting LEAF results for locations with significantly different food supply chains than France. Additional Data Sources. The current work examines the relationship between product name and environmental impact. Other (potentially unlabelled) data sources, such as ingredient lists, country of production, country of consumption, transportation method, and packaging data, could provide additional insights for more accurate environmental impact predictions. A new model that combines different data sources under varying levels of uncertainty could be superior. Processing of LCA Data. The current dataset has redundancy among certain classes. For example, there are three classes for almonds (peeled, unpeeled, and salted), all with the same LCA values. A compressed mapping for a new class distribution specific to EF score estimation could improve stability and performance by reducing the parameter count