DOI: 10.5281/zenodo.15806886
title: Cultural ecosystem service labels for photos from Flickr and Twitter using artificial intelligence models

description: Description:

Dataset of photos downloaded from Flickr (241,582 photos) and Twitter-X (1,035,488 photos) labeled by different artificial intelligence models and validated by labels assigned by human experts. 

The entire dataset was labeled using different AI models. First, we applied a Large Language Model (GPT-4.1 from OPENAI) and Llava 1.6 (on a subset of the data) to extract semantic labels from the image content based on prompts fine-tuned using prompt engineering.

In parallel, we used the base version of DINO (a self-supervised vision transformation model), fine-tuned with a subset of human expert-labeled images from our own dataset, to generate inferences for the entire image collection.

We also incorporated labels derived from expert vision models pre-trained on established datasets such as ImageNet, COCO, Places365, and Nature, which provided complementary classification information.

The labels used correspond to two categories (Table 1):

 

Table 1. Categories used in social media photo tagging: Stoten, based on the scientific framework proposed by Moreno-Llorca et al. (2020). Level 3, a hierarchical tagging system developed by our team to provide greater thematic detail, especially suited for the identification of Cultural Ecosystem Services.








Stoten



Level3





Cultural



Accommodation





Fauna/Flora



Air activities





Gastronomy



Animals





Nature & Landscape



Breakwater





Not relevant



Bridge





Recreational



Commerce facilities





Religious



Cities





Rural tourism



Clouds





Sports



Dam





Sun and beach



Dock





Urban



Fungus



 


Heritage and culture



 


Knowledge



 


Landscapes



 


Lighthouse



 


Not relevant



 


Other abiotic features



 


Plants



 


Roads



 


Shelter



 


Skies



 


Spiritual, symbolic and related connotations



 


Terrestrial activities



 


Towns and villages



 


Tracks and trails



 


Vegetation and habitats



 


Vehicle



 


Water activities



 


Wind farm



 


Winter activities






 

Table 2. Table of contents of the dataset








Folder

 
 
 


format



Description





AI models



DINO

 


model



.pt and pth



Model fine-tuned with a subset of expert-labeled images



 


Expert models



CES_label_tree



.csv



Equivalence table used to assign labels generated by expert models to our categories of interest (Stoten and Level3)



 


LLMs GPT and Llava prompts



GPT_Label_local_files



.py



Python script used for labeling photos using OPENAI models (in our case we used the GPT 4.1 model)



 
 
 


Level3_GPT_LLava_7_prompts_used



.txt



Seven prompts used for photo tagging using GPT 4.1 and Llava 1.6



 
 
 


Stoten_GPT_LLava_7_prompts_used



.txt



Seven prompts used for photo tagging with Stoten using GPT 4.1 and Llava 1.6



 
 
 


Stoten_Level3_categories



.csv



Seven prompts used for photo tagging with level 3 using GPT 4.1 and Llava 1.6





Flickr



AI based labels



DINO



Flickr_DINO_all



.csv



Inferences for all Flickr photos from the DINO model trained with the ground truth



 
 


Expert models



Flickr_expert_models_all



.csv



Labels generated by expert models for the entire database



 
 


GPT



Flickr_GPT_all



.csv



Database of Flickr photos tagged with CES using OPENAI's GPT-4.1 model.



 
 
 


Flickr_GPT_7_prompts_8192



.csv



Subset of the Flickr photo database with CES-related tags assigned by the GPT 4.1 model where 7 prompts are tested for Stoten and Level 3.



 
 


Llava 1.6



Flickr_Llava_1-6



.csv



Subset of the Flickr photo database with CES-related tags assigned by the Llava 1.6 model where 7 prompts are tested for Stoten and Level 3.



 


Ground truth



Ground Truth labels



Flickr_Database_Labeled_1082



.csv



Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 1082 Flickr photos



 
 
 


Flickr_Database_Labeled_7110



.csv



Ground Truth, an archive containing labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 7110 Flickr photos



 
 
 


Flickr_Database_Labeled_8192



.csv



Union of the two databases labeled above



 
 


Ground Truth photos



1082



.jpg/png



Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES



 
 
 


7110



.jpg/png



Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES



 


Human labels



Flickr_DataBase_Labeled_1082_expert1_AS



.csv



File containing tags assigned by expert 1 for both Stoten and Level 3, from 1082 Flickr photos



 
 
 


Flickr_DataBase_Labeled_1082_expert2_FG



.csv



File containing tags assigned by expert 2 for both Stoten and Level 3, from 1082 Flickr photos



 
 
 


Flickr_DataBase_Labeled_7110_expert1_CN



.csv



File containing tags assigned by expert 1 for both Stoten and Level 3, from 7110 Flickr photos





Twitter



AI based labels



DINO



Twitter_DINO_all



.csv



Inferences for all Twitter photos from the DINO model trained with the ground truth



 
 


Expert models



Twitter_expert_models_all



.csv



Labels generated by expert models for the entire database



 
 


GPT



Twitter_GPT_all



.csv



Database of Twitter photos tagged with CES using OPENAI's GPT-4.1 model.



 
 
 


Twitter_GPT_7_prompts_150



.csv



Subset of the Twitter photo database with CES-related tags assigned by the GPT 4.1 model where 7 prompts are tested for Stoten and Level 3.



 
 


Llava 1.6



Twitter_Llava_1-6



.csv



Subset of the Twitter photo database with CES-related tags assigned by the Llava 1.6 model where 7 prompts are tested for Stoten and Level 3.



 


Ground truth



Ground Truth labels



Twitter_Database_Labeled_150



.csv



Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 150 Twitter photos



 
 
 


Twitter_Database_Labeled_6804



.csv



Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 6804 Twitter photos



 
 


Ground Truth photos



150



.jpg/png



Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES



 
 
 


6804



.jpg/png



Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES



 


Human labels



Flickr_DataBase_Labeled_150_7experts



.csv



File containing tags assigned by 7 experts  for both Stoten and Level 3, from 150 Twitter photos



 
 
 


Flickr_DataBase_Labeled_6804_expert1_FG



.csv



File containing tags assigned by expert 2 for both Stoten and Level 3, from 6804 Twitter photos






 

References: 

Moreno-Llorca, R., Méndez, P. F., Ros-Candeira, A., Alcaraz-Segura, D., Santamaría, L., Ramos-Ridao, Á. F., ... & Vaz, A. S. (2020). Evaluating tourist profiles and nature-based experiences in Biosphere Reserves using Flickr: Matches and mismatches between online social surveys and photo content analysis. Science of the Total Environment, 737, 140067. https://doi.org/10.1016/j.scitotenv.2020.140067