## 3 Methodology
### Methodology for Post Collection and Annotation
#### 3.1.2 Annotation Procedure  
Post collection and annotation is a multi-stage process illustrated in Figure 2. The first part in this process is to collect Twitter posts from the social media platform. The streaming API was used for the continuous collection of tweets for a specific time period. Historically, one per cent of tweets posted on Twitter every day were freely accessible by researchers and they could collect tweets based on a random sampling approach. We used a filtering approach based on locations, where tweets were posted to collect data. This filter with a rectangular bounding box covering the major areas of the Gold Coast helped us to only collect tweets that originated from within the Gold Coast and surrounding suburbs. Uniform random sampling was further used to select a subset of the collected posts for the manual annotation. It is important noting that, in general, people are more positive than negative in their daily lives resulting in a natural bias toward positivity. Therefore, datasets crawled from social media networks, e.g., Twitter, are generally imbalanced. The sample data selected based on uniform random sampling for annotation and further analysis also followed a similar distribution to the crawled dataset. Coordinated by two project team members (one with tourism and the other one with information technology expertise), the annotation process was completed in two steps: training and annotation of the full dataset. Each post was annotated independently by two researchers following the instructions developed by the team of experts (Section 3.1.3), who coordinated the annotation effort and also annotated the training data. A two-stage annotation scheme was proposed and adopted to facilitate the manual annotation process. In the first stage, the selected postswere initially analysed at the post level to determine the relevance of posts to tourism (i.e., in scope). In the second stage, and for tourism-related posts, a three-level annotation approach consisting of post, sentence, and expression levels was proposed. Annotators were proficient in English and familiar with the subject of tourism.  
Annotation used a hybrid approach, combining the top-down coding process informed by the literature on key destination attributes and emotions with the bottom-up coding process through which new targets/aspects were added to the literature-informed list of attributes and emotions as they emerged in the annotation process. Annotators were provided with the hierarchical system of targets and aspects [36; 37; 38; 39] as well as the emotional vocabulary list (e.g., Your emotional vocabulary list by [PERSON], [[https://karlamclaren.com](https://karlamclaren.com)]([https://karlamclaren.com](https://karlamclaren.com))[47] (accessed on 12 April 2023) for coding. After reading the annotation instructions, they were trained on 100 Twitter posts, out of which 15 were identified as tourism related. In the training stage, annotators discussed with one another and the team of experts, and they were allowed to ask questions to clarify their understanding of the scheme. Similar training took place in the second stage with the three-level annotation of tourism-related tweets. Training improves the robustness and reliability of annotation. As reported in Section 4.1.1, the inter-annotator agreement is high among annotators. For cases of disagreement between annotators, harmonisation then followed, involving discussion of all four coordinators and annotators to reach one agreed version for each tweet in terms of all properties of the tweet.