# NLP Meets Agronomy: Document Classification for Plant Health Surveillance

Luis Antonio Vasquez Reina, Robert Bossy

► To cite this version:

Luis Antonio Vasquez Reina, Robert Bossy. NLP Meets Agronomy: Document Classification for Plant Health Surveillance. Université de Lorraine; IDMC (Univeristé de Lorraine); Institut national de recherche pour l'agriculture, l'alimentation et l'environnement (INRAE); Universidad del País Vasco / Euskal Herriko Unibertsitatea UPV/EHU. 2023. hal-04223051

HAL Id: hal-04223051

https://hal.science/hal-04223051v1

Submitted on 12 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
UNIVERSITÉ DE LORRAINE

MASTER THESIS

NLP Meets Agronomy:
Document Classification for
Plant Health Surveillance

Author:
Luis Antonio VASQUEZ REINA

Supervisors:
Robert BOSSY
Miguel COUCEIRO
Rodrigo AGERRI

Reviewer:
Marianne CLAUSEL

A thesis submitted in fulfillment of the requirements
for the degree of Master of Science in Natural Language Processing
in the

Applied Mathematics and Informatics from Genome to the Environment
Group (MaIAGE)

Institut national de recherche pour l'agriculture, l'alimentation et
l'environnement (INRAE)

Institut du Digital, Management et Cognition (IDMC)

Informatics Faculty of the University of the Basque Country (UPV-EHU)

31 August 2023
<figure><img src="image_2.png" /></figure>
Declaration of Authorship

I, Luis Antonio VASQUEZ REINA, declare that this thesis titled, "NLP Meets Agronomy: Document Classification for Plant Health Surveillance" and the work presented in it are my own. I confirm that:

* This work was done wholly or mainly while in candidature for a research degree at this University.

* Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated.

* Where I have consulted the published work of others, this is always clearly attributed.

* Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work.

* I have acknowledged all main sources of help.

* Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself.

Signed:

Date:
<figure><img src="image_3.png" /></figure>
“This thesis is dedicated to everyone who has not dedicated their thesis to themselves.”
<figure><img src="image_4.png" /></figure>
UNIVERSITÉ DE LORRAINE

Abstract

Institut du Digital, Management et Cognition (IDMC)

Informatics Faculty of the University of the Basque Country (UPV-EHU)

Institut national de recherche pour l'agriculture, l'alimentation et l'environnement
(INRAE)

Master of Science in Natural Language Processing

NLP Meets Agronomy: Document Classification for Plant Health Surveillance

by Luis Antonio VASQUEZ REINA

In the field of Plant Health Epidemiological Surveillance, accurately analyzing written reports of events affecting agriculture is crucial. This master's thesis in Natural Language Processing leverages the power of Automatic Text Classification, specifically focusing on its application in monitoring plant health. Conducted as part of the *TIERS-ESV* project, this work is a joint effort between the Bibliome team at the INRAE MaIAGE Laboratory and the VSI team at the PESV Plant Health Surveillance Platform. Central to our research is a dataset curated and annotated by the VSI team, consisting of reports extracted from online sources. This diverse, multilingual dataset was subjected to thorough preprocessing, applying methods for noise elimination, error message removal, and addressing scrapping errors and annotation discrepancies. We employed various BERT models for Text Classification, tailored to our dataset via Fine-tuning and pattern-based training method. After extensive classifier training, we selected the top-performing models. The efficacy of the models obtained as a result of this work will result in the deployment of BERT-based classifiers, poised to assist the VSI experts in their monitoring mission.
Resumé

Dans le domaine de la Surveillance Épidémiologique de la Santé des Plantes, l'anal-
yse précise des rapports écrits sur les événements affectant l'agriculture est cruciale.
Ce mémoire de maîtrise en Traitement Automatique des Langues exploite les capacités
de Classification Automatique des Textes, en se concentrant sur son application dans
la surveillance de la santé des plantes. Réalisé dans le cadre du projet TIERS-ESV ce
travail est le fruit d'une collaboration entre l'équipe Bibliome de l'unité INRAE Ma-
IAGE et l'équipe VSI de la Plateforme de Surveillance de la Santé des Plantes PESV.
Notre recherche s'appuie sur un ensemble de données curées et annotées par l'équipe
VSI, qui consiste en des rapports extraits de sources collectées en ligne. Cet ensemble
de données diversifié et multilingue a été soumis à un prétraitement approfondi, en
appliquant des méthodes d'élimination du bruit, de suppression des messages d'erreur
et de traitement des erreurs de mise au rebut et des divergences d'annotation. Nous
avons utilisé divers modèles BERT pour la Classification des Textes, adaptés à notre
ensemble de données par le biais d'un finetuning et l'Apprentissage par Exploitation
de Motifs. Après un entraînement intensif à la classification, nous avons sélectionné
les modèles les plus performants. L'efficacité des modèles créés dans le cadre de ce
travail conduira à l'utilisation de classificateurs basés sur BERT pour aider les experts
de l'équipe VSI dans leur tâche de surveillance.
*Acknowledgements*

Starting this Master's degree has been a great personal step. Moving abroad and really learning about other cultures is a both an opportunity and a challenge. Facing completely different expectations around social norms, education, and bureaucracy has been a test.

I appreciate everyone that supported me during this journey, especially, I would like to thank, ordered by name length and then alphabetically, Anna, Flor, Diego, Elisa, María, Shane, Isabel, Joanna, Karolin, Maxime, William, and Luis Felipe. Additional thanks go to the LCT students and people that made our time more enjoyable.

I also thank my supervisors for the helpful discussions and support they offered me for this thesis, and the Saclay-IA platform for the computational resources provided.

Finally, I would like to acknowledge the LCT Consortium for putting together this Erasmus Mundus Master Program, and especially for the scholarship I received, without which, pursuing this degree would not have been feasible.
<figure><img src="image_5.png" /></figure>
# Contents

<table><tr><td>Declaration of Authorship</td><td>iii</td></tr><tr><td>Abstract</td><td>vii</td></tr><tr><td>Acknowledgements</td><td>ix</td></tr><tr><td>1 Introduction</td><td>1</td></tr><tr><td>1.1 Motivation</td><td>1</td></tr><tr><td>1.2 Hosting Institution: INRAE Laboratory</td><td>1</td></tr><tr><td>1.3 Overarching Project: TIERS-ESV</td><td>2</td></tr><tr><td>1.4 Objective</td><td>2</td></tr><tr><td>1.5 Structure of this Thesis</td><td>3</td></tr><tr><td>2 Background</td><td>5</td></tr><tr><td>2.1 Text Classification</td><td>5</td></tr><tr><td>2.2 Neural Networks</td><td>5</td></tr><tr><td>2.2.1 Motivating Example</td><td>6</td></tr><tr><td>2.2.2 Neurons</td><td>6</td></tr><tr><td>2.2.3 Activation functions</td><td>7</td></tr><tr><td>2.2.4 Loss Functions</td><td>8</td></tr><tr><td>2.2.5 Backpropagation</td><td>8</td></tr><tr><td>2.2.6 Training</td><td>9</td></tr><tr><td>2.2.7 Overfitting</td><td>9</td></tr><tr><td>2.2.8 Optimizers</td><td>11</td></tr><tr><td>2.2.9 Feed-Forward Neural Networks</td><td>12</td></tr><tr><td>2.2.10 Hyperparameters</td><td>13</td></tr><tr><td>2.2.11 Neural Networks for NLP tasks</td><td>13</td></tr><tr><td>2.3 Tokenization and Embeddings</td><td>14</td></tr><tr><td>2.4 BERT Models for Text Classification</td><td>16</td></tr><tr><td>2.4.1 Attention and Transformers</td><td>17</td></tr><tr><td>2.4.2 The training of BERT</td><td>18</td></tr><tr><td>Masked Language Modeling</td><td>19</td></tr><tr><td>Next Sentence Prediction</td><td>19</td></tr><tr><td>2.4.3 Using BERT for Text Classification</td><td>19</td></tr><tr><td>2.5 Evaluation Metrics for Classification</td><td>20</td></tr><tr><td>3 Related Work</td><td>23</td></tr><tr><td>4 The VSI Dataset</td><td>25</td></tr><tr><td>4.1 Dataset Collection</td><td>25</td></tr><tr><td>4.2 Dataset Annotation</td><td>26</td></tr><tr><td>4.3 Dataset Statistics</td><td>31</td></tr><tr><td>4.4 Dataset Issues</td><td>33</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>4.4.1 Duplicate Entries</td>
      <td>33</td>
    </tr>
    <tr>
      <td>4.4.2 Scrapping Failures</td>
      <td>33</td>
    </tr>
    <tr>
      <td>4.4.3 Scrapping errors</td>
      <td>34</td>
    </tr>
    <tr>
      <td>4.4.4 Data Noise</td>
      <td>36</td>
    </tr>
    <tr>
      <td>4.4.5 Annotation Inconsistencies</td>
      <td>36</td>
    </tr>
    <tr>
      <td><b>5 Preprocessing the VSI dataset</b></td>
      <td><b>41</b></td>
    </tr>
    <tr>
      <td>5.1 Leveraging Keywords</td>
      <td>41</td>
    </tr>
    <tr>
      <td>5.2 Data Cleaning</td>
      <td>42</td>
    </tr>
    <tr>
      <td>5.2.1 Noise Removal</td>
      <td>42</td>
    </tr>
    <tr>
      <td>5.2.2 Deleting Error Messages</td>
      <td>43</td>
    </tr>
    <tr>
      <td>5.2.3 Handling scrapping errors</td>
      <td>43</td>
    </tr>
    <tr>
      <td>5.3 Resolving Inconsistencies and Duplicate Documents</td>
      <td>44</td>
    </tr>
    <tr>
      <td>5.4 Preprocessing Results</td>
      <td>46</td>
    </tr>
    <tr>
      <td><b>6 Methodology</b></td>
      <td><b>51</b></td>
    </tr>
    <tr>
      <td>6.1 BERT Models</td>
      <td>51</td>
    </tr>
    <tr>
      <td>6.1.1 BERT-base</td>
      <td>51</td>
    </tr>
    <tr>
      <td>6.1.2 mBERT</td>
      <td>51</td>
    </tr>
    <tr>
      <td>6.1.3 Bio-Link-BERT</td>
      <td>51</td>
    </tr>
    <tr>
      <td>6.1.4 SciBERT</td>
      <td>52</td>
    </tr>
    <tr>
      <td>6.1.5 RoBERTa-base</td>
      <td>52</td>
    </tr>
    <tr>
      <td>6.1.6 XLM-RoBERTa</td>
      <td>52</td>
    </tr>
    <tr>
      <td>6.1.7 Excluded Models</td>
      <td>52</td>
    </tr>
    <tr>
      <td>6.2 Fine-tuning</td>
      <td>53</td>
    </tr>
    <tr>
      <td>6.3 Pattern-Exploiting Training</td>
      <td>54</td>
    </tr>
    <tr>
      <td>6.4 Splitting the Dataset</td>
      <td>56</td>
    </tr>
    <tr>
      <td>6.5 Implementation, Hyperparameters, and Hardware</td>
      <td>57</td>
    </tr>
    <tr>
      <td>6.6 Methodology Summary</td>
      <td>58</td>
    </tr>
    <tr>
      <td><b>7 Results and Discussion</b></td>
      <td><b>61</b></td>
    </tr>
    <tr>
      <td><b>8 Practical Implications and Future Work</b></td>
      <td><b>67</b></td>
    </tr>
    <tr>
      <td>8.1 Overview</td>
      <td>67</td>
    </tr>
    <tr>
      <td>8.2 Inference Service</td>
      <td>67</td>
    </tr>
    <tr>
      <td>8.3 Training Service</td>
      <td>68</td>
    </tr>
    <tr>
      <td>8.4 Consequences for Plant Health Surveillance</td>
      <td>68</td>
    </tr>
    <tr>
      <td>8.5 Future Work</td>
      <td>70</td>
    </tr>
    <tr>
      <td><b>9 Conclusions</b></td>
      <td><b>71</b></td>
    </tr>
    <tr>
      <td><b>Bibliography</b></td>
      <td><b>73</b></td>
    </tr>
    <tr>
      <td><b>A Evaluation Metrics for Classification</b></td>
      <td><b>79</b></td>
    </tr>
    <tr>
      <td>A.1 Evaluating Probabilistic Classifiers</td>
      <td>80</td>
    </tr>
    <tr>
      <td><b>B VSI Dataset samples and language statistics</b></td>
      <td><b>83</b></td>
    </tr>
    <tr>
      <td>B.1 Keywords for constructing the VSI Database</td>
      <td>83</td>
    </tr>
    <tr>
      <td>B.2 Sample entries detected as Latin</td>
      <td>85</td>
    </tr>
    <tr>
      <td>B.3 VSI Language statistics</td>
      <td>86</td>
    </tr>
  </tbody>
</table>
<table><tr><td>C Regular expressions used in this project</td><td>87</td></tr><tr><td>C.1 Error Message Removal - VSI Dataset</td><td>87</td></tr><tr><td>C.2 Noise Removal - VSI Dataset</td><td>87</td></tr><tr><td>D Preprocessing Statistics</td><td>89</td></tr><tr><td>D.1 Preprocessing the VSI dataset</td><td>89</td></tr><tr><td>E Training Setup</td><td>91</td></tr><tr><td>E.1 GPU Specifications</td><td>91</td></tr><tr><td>E.2 Statistics for Splits</td><td>92</td></tr><tr><td>F Training Results</td><td>95</td></tr><tr><td>F.1 VSI Optimal Epochs</td><td>95</td></tr><tr><td>F.2 VSI Metrics at Optimal Epochs</td><td>97</td></tr></table>
<figure><img src="image_6.png" /></figure>
List of Figures

<table>
  <tbody>
    <tr>
      <td>2.1</td>
      <td>Basic Neuron</td>
      <td>7</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>Common activation functions</td>
      <td>7</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>Gradient descent illustration</td>
      <td>9</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>Neuron Training Process</td>
      <td>10</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>Evolution of loss during training</td>
      <td>10</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Evolution of loss on the training and development splits</td>
      <td>11</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>A Feed-Forward Neural Network</td>
      <td>12</td>
    </tr>
    <tr>
      <td>2.8</td>
      <td>Neural Networks for Classification Tasks</td>
      <td>13</td>
    </tr>
    <tr>
      <td>2.9</td>
      <td>Naive tokenization</td>
      <td>14</td>
    </tr>
    <tr>
      <td>2.10</td>
      <td>Different tokenization approaches: naive, Byte-Pair Encoding, and Word Piece</td>
      <td>15</td>
    </tr>
    <tr>
      <td>2.11</td>
      <td>Word and Document embeddings</td>
      <td>15</td>
    </tr>
    <tr>
      <td>2.12</td>
      <td>Using Embeddings to train Neural Networks for Classification</td>
      <td>16</td>
    </tr>
    <tr>
      <td>2.13</td>
      <td>Attention Mechanism for Machine Translation, from Olah and Carter (2016)</td>
      <td>17</td>
    </tr>
    <tr>
      <td>2.14</td>
      <td>Classical Transformer Architecture</td>
      <td>18</td>
    </tr>
    <tr>
      <td>2.15</td>
      <td>Self-Attention for MLM</td>
      <td>19</td>
    </tr>
    <tr>
      <td>2.16</td>
      <td>Next Sentence Prediction Task for BERT</td>
      <td>20</td>
    </tr>
    <tr>
      <td>2.17</td>
      <td>Using BERT Embeddings for Document Classification</td>
      <td>20</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>VSI Data Collection Pipeline</td>
      <td>25</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>VSI Annotation Interface for Articles</td>
      <td>29</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>VSI Annotation Interface for Subjects</td>
      <td>30</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Distribution of subjects in the VSI dataset without preprocessing</td>
      <td>31</td>
    </tr>
    <tr>
      <td>4.5</td>
      <td>Language distribution of entries in the VSI dataset</td>
      <td>32</td>
    </tr>
    <tr>
      <td>4.6</td>
      <td>Unique entries per source of content in the VSI dataset</td>
      <td>32</td>
    </tr>
    <tr>
      <td>4.7</td>
      <td>Length (in n. of tokens) of the Content Sources in the VSI dataset</td>
      <td>33</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Extracting phrases with Keywords</td>
      <td>41</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Unique entries per source of content in the VSI dataset</td>
      <td>42</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>Removing noise from multilingual strings</td>
      <td>43</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Unique Entries per content source before and after preprocessing</td>
      <td>46</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>Length (in n. of tokens) of the Content Sources in the VSI dataset after preprocessing</td>
      <td>48</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>Positive/Negative balance after preprocessing the VSI dataset</td>
      <td>49</td>
    </tr>
    <tr>
      <td>6.1</td>
      <td>Fine-tuning BERT for Document Classification</td>
      <td>53</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>Intuition behind PET</td>
      <td>54</td>
    </tr>
    <tr>
      <td>6.3</td>
      <td>Training a PET Ensemble</td>
      <td>56</td>
    </tr>
    <tr>
      <td>6.4</td>
      <td>Creating a Synthetically Labeled Dataset with the PET Ensemble, and subsequently Fine-tuning</td>
      <td>56</td>
    </tr>
    <tr>
      <td>6.5</td>
      <td>Strategies for Dataset Splitting</td>
      <td>57</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>7.1</td>
      <td>Unbalanced Fine-tuning - Losses and Development F<sub>2</sub> evolution for mBERT on the Title</td>
      <td>61</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>PET 1000 - Losses and Development F<sub>2</sub> evolution for RoBERTa-base on the Title</td>
      <td>61</td>
    </tr>
    <tr>
      <td>7.3</td>
      <td>Development F<sub>2</sub> for mBERT at the optimal epochs, across all the training methods</td>
      <td>62</td>
    </tr>
    <tr>
      <td>7.4</td>
      <td>Sample AUCs for some classifiers</td>
      <td>63</td>
    </tr>
    <tr>
      <td>8.1</td>
      <td>Our classifiers added to the VSI Data Collection Pipeline</td>
      <td>67</td>
    </tr>
    <tr>
      <td>A.1</td>
      <td>Area Under Curve</td>
      <td>82</td>
    </tr>
  </tbody>
</table>
List of Tables

<table><tbody><tr><td>2.1</td><td>Common activation functions</td><td>8</td></tr><tr><td>2.2</td><td>Common optimization methods for backpropagation</td><td>12</td></tr><tr><td>4.1</td><td>Sources of text content in the VSI Dataset</td><td>26</td></tr><tr><td>4.2</td><td>Sample entries from the VSI dataset</td><td>28</td></tr><tr><td>4.3</td><td>Some error messages in the VSI dataset</td><td>34</td></tr><tr><td>4.4</td><td>Some headers parsed by Trafilatura</td><td>34</td></tr><tr><td>4.5</td><td>Some website names parsed by Trafilatura</td><td>34</td></tr><tr><td>4.6</td><td>Inconsistencies introduced by scrapping errors</td><td>35</td></tr><tr><td>4.7</td><td>Noise in the VSI dataset</td><td>37</td></tr><tr><td>4.8</td><td>Similar entries with website names as a suffix</td><td>38</td></tr><tr><td>4.9</td><td>Annotation inconsistencies in the VSI dataset</td><td>39</td></tr><tr><td>5.1</td><td>Sources of text content in the VSI Dataset after preprocessing</td><td>42</td></tr><tr><td>5.2</td><td>Error Message Count per Content Source in the VSI dataset</td><td>43</td></tr><tr><td>5.3</td><td>Short Message Count per Content Source in the VSI dataset</td><td>44</td></tr><tr><td>5.4</td><td>Determining Relevance for Title</td><td>45</td></tr><tr><td>5.5</td><td>Unique Entries per content source before and after preprocessing</td><td>46</td></tr><tr><td>5.6</td><td>Mean token counts before and after preprocessing the VSI dataset</td><td>47</td></tr><tr><td>6.1</td><td>Sizes of BERT models used for this project. Adapted from Conneau et al. (2020)</td><td>51</td></tr><tr><td>6.2</td><td>Patterns and Verbalizers used for PET</td><td>55</td></tr><tr><td>6.3</td><td>Code repositories for this work</td><td>57</td></tr><tr><td>6.4</td><td>Hyperparameters</td><td>58</td></tr><tr><td>6.5</td><td>Approximate Training times per Content Source, in GPU hours</td><td>59</td></tr><tr><td>7.1</td><td>Best classifiers by Content Source</td><td>64</td></tr><tr><td>8.1</td><td>Positives and FN ratios for the best classifiers</td><td>69</td></tr><tr><td>A.1</td><td>Classification Outcomes</td><td>79</td></tr><tr><td>A.2</td><td>Confusion Matrix for Binary Classification</td><td>79</td></tr><tr><td>B.1</td><td>Keywords and key phrases used for the VSI dataset</td><td>84</td></tr><tr><td>B.2</td><td>Sample entries detected as Latin text</td><td>85</td></tr><tr><td>B.3</td><td>Number of entries per language in the VSI Dataset</td><td>86</td></tr><tr><td>C.1</td><td>Some Regular Expressions for Removing Errors Messages in the VSI dataset</td><td>87</td></tr><tr><td>C.2</td><td>Regular Expressions for Noise Removal</td><td>88</td></tr><tr><td>D.1</td><td>Unique Entries per content source before and after preprocessing</td><td>89</td></tr><tr><td>D.2</td><td>Token Length statistics after preprocessing the VSI dataset</td><td>89</td></tr><tr><td>D.3</td><td>Label Distribution after preprocessing the VSI dataset</td><td>89</td></tr></tbody></table>
<table>
  <tbody>
    <tr>
      <td>E.1 Specifications of Lab-IA Nodes</td>
      <td>91</td>
    </tr>
    <tr>
      <td>E.2 Split Statistics for Unbalanced Fine-tuning</td>
      <td>92</td>
    </tr>
    <tr>
      <td>E.3 Split Statistics for Balanced Fine-tuning</td>
      <td>92</td>
    </tr>
    <tr>
      <td>E.4 Split Statistics for PET 50</td>
      <td>92</td>
    </tr>
    <tr>
      <td>E.5 Split Statistics for PET 100</td>
      <td>93</td>
    </tr>
    <tr>
      <td>E.6 Split Statistics for PET 200</td>
      <td>93</td>
    </tr>
    <tr>
      <td>E.7 Split Statistics for PET 500</td>
      <td>93</td>
    </tr>
    <tr>
      <td>E.8 Split Statistics for PET 1000</td>
      <td>93</td>
    </tr>
    <tr>
      <td>F.1 Optimal epochs for Unbalanced Fine-tuning</td>
      <td>95</td>
    </tr>
    <tr>
      <td>F.2 Optimal epochs for Balanced Fine-tuning</td>
      <td>96</td>
    </tr>
    <tr>
      <td>F.3 Optimal epochs for PET 50</td>
      <td>96</td>
    </tr>
    <tr>
      <td>F.4 Optimal epochs for PET 100</td>
      <td>96</td>
    </tr>
    <tr>
      <td>F.5 Optimal epochs for PET 200</td>
      <td>96</td>
    </tr>
    <tr>
      <td>F.6 Optimal epochs for PET 500</td>
      <td>96</td>
    </tr>
    <tr>
      <td>F.7 Optimal epochs for PET 1000</td>
      <td>96</td>
    </tr>
    <tr>
      <td>F.8 BERT-base F<sub>2</sub> on Development split</td>
      <td>98</td>
    </tr>
    <tr>
      <td>F.9 mBERT F<sub>2</sub> on Development split</td>
      <td>98</td>
    </tr>
    <tr>
      <td>F.10 Bio-Link-BERT F<sub>2</sub> on Development split</td>
      <td>99</td>
    </tr>
    <tr>
      <td>F.11 SciBERT F<sub>2</sub> on Development split</td>
      <td>99</td>
    </tr>
    <tr>
      <td>F.12 RoBERTa-base F<sub>2</sub> on Development split</td>
      <td>100</td>
    </tr>
    <tr>
      <td>F.13 XLM-RoBERTa F<sub>2</sub> on Development split</td>
      <td>100</td>
    </tr>
    <tr>
      <td>F.14 BERT-base AUC on Development split</td>
      <td>101</td>
    </tr>
    <tr>
      <td>F.15 mBERT AUC on Development split</td>
      <td>101</td>
    </tr>
    <tr>
      <td>F.16 Bio-Link-BERT AUC on Development split</td>
      <td>102</td>
    </tr>
    <tr>
      <td>F.17 SciBERT AUC on Development split</td>
      <td>102</td>
    </tr>
    <tr>
      <td>F.18 RoBERTa-base AUC on Development split</td>
      <td>103</td>
    </tr>
    <tr>
      <td>F.19 XLM-RoBERTa AUC on Development split</td>
      <td>103</td>
    </tr>
    <tr>
      <td>F.20 Development F<sub>2</sub> and AUC of top performing classifiers by BERT Model</td>
      <td>104</td>
    </tr>
    <tr>
      <td>F.21 Development F<sub>2</sub>, AUC, and F<sub>1</sub> of top performing classifiers by Content Source</td>
      <td>105</td>
    </tr>
    <tr>
      <td>F.22 Development and Test metrics for Best Performing classifiers</td>
      <td>106</td>
    </tr>
  </tbody>
</table>
Abbreviations

SoftMax Soft-ArgMax. 8, 13, 19, 55

$\sigma$ Logistic function. 8

tanh Hyperbolic Tangent. 8, 19

AI Artificial Intelligence. 1, 2, 14, 23

AUC Area Under the Curve. 57, 81, 82

BERT Bidirectional Encoder Representations from Transformers. 16–20, 23, 51–55, 57, 58, 62–65, 71

BLURB Biomedical Language Understanding and Reasoning Benchmark. 52

BPE Byte-Pair Encoding. 14, 15, 52

FFNN Feed-Forward Neural Network. 12

ML Machine Learning. 2, 5, 7, 12, 15, 54

MLM Masked Language Modelling. 19, 54, 55

NLP Natural Language Processing. 1, 2, 5, 6, 13–15, 17, 23, 53, 54, 71

PESV Plateforme d'Epidémiosurveillance en Santé Végétale. 2, 25, 27, 57, 67, 68, 71

PET Pattern-Exploiting Training. 54–59, 61, 62, 64, 65, 71

ReLU Rectified Linear Unit. 8

RNN Recurrent Neural Network. 16

TIERS-ESV Traitement de l'Information et Expertise des Risques Sanitaires pour l'Epidémiosurveillance en Santé Végétal. 2, 67

VSI Veille Sanitaire Internationale. 2, 25–27, 33, 36, 41, 44, 67–71

WP Word Piece. 14, 18, 51, 52
<figure><img src="image_7.png" /></figure>
# Chapter 1

# Introduction

## 1.1 Motivation

Natural Language Processing (NLP) is a subfield of Artificial Intelligence (AI) that focuses on the interaction between computers and humans through natural language. Its primary aim is to enable computers to understand, interpret, and generate human language in a way that is both meaningful and useful. Recently, NLP has been revolutionized by the widespread adoption of Neural Networks. As in many domains, AI and NLP have been starting to be adopted for Biomedical purposes, with promising results (Alrowili and Shanker, 2021; Hakala and Pyysalo, 2019).

In this thesis, we present an application of Text Classification to Plant Health Surveillance, which refers to the systematic observation, detection, and analysis of plant diseases and pests to prevent their spread and ensure the health and productivity of plants. It involves tracking disease patterns, assessing risks, and implementing measures to protect plant ecosystems and agricultural systems.

## 1.2 Hosting Institution : INRAE Laboratory

INRAE<fnref n="1" /> (Institut National de Recherche pour l'Agriculture, l'Alimentation et l'Environnement) is a French research institute dedicated to agriculture, food, and the environment.

INRAE conducts scientific research and innovation activities to address various challenges related to sustainable agriculture, food production, and environmental conservation. Its research efforts span a wide range of disciplines, including agronomy, biology, ecology, genetics, forestry, hydrology, and applied mathematics and computer science.

Through its multidisciplinary approach and emphasis on sustainability, INRAE plays a crucial role in driving scientific progress, innovation, and policy development in France and beyond, with the ultimate aim of ensuring sustainable and resilient agricultural and environmental systems.

Within INRAE, the Mathematics and Numerics division (MathNum<fnref n="2" />) focuses on advancing research in various fields including applied mathematics, statistics, bioinformatics, AI, and information technology. The division's research units engage in theoretical, methodological, and applied research. They actively collaborate with teams from different divisions within INRAE and external organizations, fostering interdisciplinary partnerships. MaiAGE<fnref n="3" />, one of these research units, is associated to the Paris-Saclay University, and brings together mathematicians, computer scientists,

<footnote n="1">¹https://www.inrae.fr/</footnote>

<footnote n="2">²https://www.inrae.fr/departements/mathnum</footnote>

<footnote n="3">³https://maiage.mathnum.inrae.fr</footnote>
bioinformaticians, and biologists to address challenges in the fields of biology, agronomy, and ecology. This project was carried out with alongside the Bibliome<fnref n="4" /> team, which specializes in the advancement and adaptation of NLP and Machine Learning (ML) techniques tailored for textual data within the fields of biology and agronomy.

## 1.3 Overarching Project: TIERS-ESV

The Bibliome team, as part of the *Traitement de l'Information et Expertise des Risques Sanitaires pour l'Epidémiosurveillance en Santé Végétal* (TIERS-ESV) project<fnref n="5" /> (*Processing of Health Risk Information and Knowledge for Epidemiological Surveillance in Plant Health*), collaborates frequently with the *Plateforme d'Epidémiosurveillance en Santé Végétale* (PESV)<fnref n="6" /> (*Plant Health Epidemiological Surveillance Platform*), and it is within this collaboration that we conducted this work.

The PESV Platform was created in 2018 by 7 French national public and private actors (INRAE, the Anses Laboratory<fnref n="7" />, the French Ministry of Agriculture, the Cirad Center<fnref n="8" />, the Acta research association <fnref n="9" />, the French Chamber of Agriculture <fnref n="10" />, and the FREDON Network<fnref n="11" />) with the objective to ensure the efficiency of epidemiological surveillance in plant health (Ministère de L'Agriculture et de L'Alimentation, 2018).

According to its mission statement, ‘the PESV Platform leverages its scientific and technical expertise to engage in three key areas: monitoring, analysis, and advisory services. Its services cater to both public policies and all professionals within the plant health sector’.

As an integral part of its monitoring endeavors, the PESV Platform includes the Veille Sanitaire Internationale (VSI)<fnref n="12" /> (*International Health Monitoring*) project. The main focus of the VSI team is “to address potential threats that may impact plant health by engaging in monitoring activities on a global scale”. Its responsibilities encompass ongoing surveillance of events like notifications, reports, and changes in surveillance strategies across the globe. Furthermore, it conducts scientific monitoring on diverse subjects, including phylogeography, spatial distributions of species, habitat suitability assessments, innovative disease prevention measures, as well as surveillance and control methodologies.

## 1.4 Objective

Our objective is to leverage AI to support the Plant Health Monitoring efforts of the PESV Platform.

The primary objective of this project is to facilitate and enhance the monitoring of Plant Health at the PESV Platform by introducing a degree of automation to the traditionally manual processes. Recognizing the critical importance of timely and accurate Plant Health monitoring, our aim is to harness the data provided to us by experts at the VSI team in order to develop a detection system based on AI. By doing

<footnote n="4">⁴https://maiage.mathnum.inrae.fr/fr/bibliome</footnote>

<footnote n="5">⁵https://plateforme-esv.fr/node/24638 and https://maiage.inrae.fr/index.php/fr/node/2111</footnote>

<footnote n="6">⁶https://plateforme-esv.fr/</footnote>

<footnote n="7">⁷https://www.anses.fr/fr</footnote>

<footnote n="8">⁸https://www.cirad.fr/</footnote>

<footnote n="9">⁹https://www.acta.asso.fr/</footnote>

<footnote n="10">¹⁰https://chambres-agriculture.fr/</footnote>

<footnote n="11">¹¹https://fredon.fr/</footnote>

<footnote n="12">¹²https://plateforme-esv.fr/thematiques/GTVSI</footnote>
so, we aspire to offer a more efficient, data-driven approach to detect early signs of plant diseases, pests, or any other anomalies. This would not only ensure a faster response and intervention but also alleviates the workload of professionals in the field. This endeavor is not merely about replacing human intervention but assisting it. By automating certain repetitive and time-consuming tasks, we aim to alleviate experts to focus on more nuanced and complex aspects of plant health, ensuring that their expertise is utilized where it is most needed.

**1.5 Structure of this Thesis**

After this Introduction, Chapter 2 provides foundational knowledge and context for understanding our work. Chapter 3 offers a review of existing literature, highlighting previous methodologies and their gaps. In Chapter 4 we delve into the Dataset, detailing its sources and characteristics, and then continue to Chapter 5, which elucidates the preprocessing techniques employed. Chapter 6 outlines the research methods and tools used, leading to Chapter 7, where our findings are presented and analyzed. Chapter 8 sheds light on the real-world relevance of the findings, and the study culminates with Chapter 9, summarizing key takeaways. Additionally, an Appendix is provided, rich with tables and detailed statistics, serving as a valuable resource for reference.

This thesis is submitted in fulfillment of the requirements for the Erasmus Mundus Language And Communication Technology<fnref n="13" /> from the European Union, as a student assigned to the University of Lorraine (France), and the University of the Basque Country (Spain).

<footnote n="13">¹³https://lct-master.org/</footnote>
<figure><img src="image_8.png" /></figure>
# Chapter 2

## Background

This Chapter introduces the conceptual tools necessary to understand our project, our methodology, and our results.

### 2.1 Text Classification

Text Classification is a fundamental task in NLP that involves automatically assigning predefined categories or labels to text documents based on their content. The goal of Text Classification is to develop computational models that can accurately classify text documents into predefined categories, enabling automated organization, retrieval, and analysis of large volumes of textual data.

Formally, given a set $V$ called the *Vocabulary*, and a set $C$ of *categories*, a *Text Classification System* is a function

$$f: V^* \times C \to \{\text{True, False}\}$$

where $V^*$ is the set of finite sequences of elements of $V$, that is, the set of *documents*. While the general definition of Text Classification accommodates multi-label classification scenarios, the specific focus of this project is on *Binary Text Classification*. In this particular case, the classification task involves assigning documents to one of two mutually exclusive categories (Shen, 2009; Costa, Oliveira, and Fileto, 2023).

In our particular scenario, we will be handling entire documents for analysis rather than, for example, individual words or proper names. Thus, the task is termed *Document Classification*. In the context of Document Classification, the fundamental objective is to leverage a training set, comprising a collection of training documents $(d_i, c_i)$, and employ a learning method or algorithm to derive a *Classifier* $\gamma$. The primary purpose of this classifier is to establish a mapping from documents to two classes. Here, for simplicity, we refer to them as the positive and negative classes. This is denoted as

$$\gamma: V^* \to \{\text{Positive, Negative}\}$$

### 2.2 Neural Networks

Neural Networks are essential tools in ML and NLP. Given their complexity, this section introduces core concepts. Starting with a practical example, we introduce the concept of 'neuron', and then we proceed to explain neural 'training' based on data, its common challenges, and the solutions used in our work. Subsequently, our exposition continues to the organization and interconnection of neurons within a structure or
architecture referred to as a Neural Network. Finally, the use of Neural Networks in NLP is also examined. We draw on the Open Access material from Zhang et al. (2023) and Wolfram (2023).

### 2.2.1 Motivating Example

To understand the rationale behind using Neural Networks, let's consider a scenario where we aim to determine how the temperature and humidity of a place relate to the perceived temperature by living beings, such as humans. In other words, given these two metrics across various locations, our goal is to forecast the felt temperature or Heat Index for each site. A simplified version of the model from Steadman (1979), can provide us with an equation for the Heat Index (HI).

$$HI = w_{Temp}T + w_{Hum}H + c$$

The main idea behind this model is that the heat index, our target, can be calculated as a weighted sum of the temperature T and the humidity H. The influence of T on the HI is given by $w_{Temp}$ and that of H by $w_{Hum}$, and we correct by a constant value c.

In this model, H and T are termed the features of a location, serving as its defining characteristics. Essentially, locations with identical features will yield the same output. The terms $w_{Temp}$ and $w_{Hum}$ are referred to as weights or parameters. They signify the relevance of the features to the model. For instance, if $w_{Hum}$ is zero, the HI would solely be influenced by T. Additionally, c is labeled as the bias or offset, representing the output value when the features are null. The equation's right-hand side, $w_{Temp}T + w_{Hum}H + c$, is recognized as the model's output or prediction.

In practice, the values for the weights and bias of this model have been determined to be:

$$ 
\begin{aligned}
w_{Temp} &= 1.1 \\
w_{Hum} &= 0.047F^{\circ} \\
c &= -10.3F^{\circ}
\end{aligned}
$$

Yet, this basic representation has its limitations. Interestingly, by introducing an additional computational step, we can achieve more accurate results. Let's consider incorporating a minor squared term through the function $f(u) = u + 0.0002u^2$. With this, our model evolves to:

$$HI = f(w_{Temp}T + w_{Hum}H + c)$$

In this enhanced model, f is termed the activation function. The model's prediction or output is derived by applying f to the weighted sum. This added computation step allows for a more authentic representation of how living beings respond to significant shifts in temperature and humidity. In general, this kind of model allows us to use numerical features, to obtain a prediction for a variable of interest.

### 2.2.2 Neurons

When dealing with increasingly complex phenomena, it becomes challenging to explicitly define the parameters for a model. This requires a broader approach. In the
realm of Machine Learning, the concept of neurons serves as a generalization of the model described earlier.

<figure><img src="image_9.png" /><figcaption>FIGURE 2.1: Basic Neuron</figcaption></figure>

Consider a model where the inputs encompass $n$ distinct features, represented as $(x_1, x_2, \dots, x_n)$. The diagram depicted in Figure 2.1 represents the following equation:

$$z = f(w_1x_1 + w_2x_2 + \dots + w_Dx_D + b)$$

Here, $w_1, w_2, \dots, w_D$ denote the weights, $b$ signifies the bias, $f$ is the activation function, and $z$ stands for the prediction.

One of the main insights in ML is that one does not need to manually define the parameters $(w_1, w_2, \dots, w_D, b)$ for this kind of model. Instead, with ample data comprising input features and observed target values, it is feasible to determine the best parameter values. This ensures the model's predictions closely align with the observed target values. Once established, this model can be applied to new data. The process of finding the optimal parameters is called *training*.

In order to find the optimal parameters, we first need to decide how to produce our final predictions, how to measure how well the model is performing, and how to update the model's parameters.

### 2.2.3 Activation functions

The method used to derive our final predictions is contingent on the specific problem to be addressed. Various activation functions cater to different problems. For instance, in *regression*, where our objective is to compute a continuous quantity, like in the example in Chapter 2.2.1, we may handle values that can range from extremely small to exceedingly large. On the other hand, classification tasks aim to predict a categorical label, either by taking a final decision or by producing probabilities for each category.

Table 2.1 and Figure 2.2 enumerate some of the most common activation functions.

<figure><img src="image_10.png" /><figcaption>FIGURE 2.2: Common activation functions</figcaption></figure>
<table><caption>TABLE 2.1: Common activation functions</caption><thead><tr><th>Activation Function</th><th>Description</th></tr></thead><tbody><tr><td>Identity</td><td>Returns input as-is</td></tr><tr><td>Rectified Linear Unit (ReLU)</td><td>Retains positive values, sets negatives to zero</td></tr><tr><td>Logistic function (σ)</td><td>Maps input to a value between 0 and 1</td></tr><tr><td>Soft-ArgMax (SoftMax)</td><td>Converts input into a probability distribution over multiple classes</td></tr><tr><td>Hyperbolic Tangent (tanh)</td><td>Maps input to a value between -1 and 1</td></tr></tbody></table>

### 2.2.4 Loss Functions

Loss functions quantify the deviation of predicted values from actual target values. The goal during training is to adjust the parameters to minimize this deviation. The process of inputting data into the neuron to compute the loss is termed the *forward pass*.

For regression tasks, the squared error is frequently employed. It calculates the squared difference between the predicted and actual values:

$$ loss_{SE}(x, w, b) = \frac{1}{2}(z_{pred}(x, w, b) - z_{real})^2 \quad (2.1) $$

where $z_{pred} = f(xw + b)$.

In classification tasks, the *cross-entropy* loss is common. The essence of cross-entropy is to maximize the model's confidence in its category assignments<fnref n="1" />. Given features $x = x_1, x_2, ..., x_n$ and predicted probabilities $p_1, p_2, ..., p_k$ for categories $c_1, c_2, ..., c_k$, and representing actual categories as $(q_1, q_2, ..., q_k)$ where $q_j = 1$ for category $j$ and $q_j = 0$ otherwise, cross entropy is:

$$ loss_{CE}(x, w, b) = - \sum_{j} q_{j}(x) \log(p_{j}(x, w, b)) \quad (2.2) $$

For datasets with uneven category distribution, or *unbalanced* datasets, a modified version called *weighted cross-entropy* can be used. It adjusts the cross-entropy based on category representation. If category $j$ has a weight of $r_j$, then:

$$ loss_{WCE}(x, w, b) = - \sum_{j} r_{j} q_{j}(x) \log(p_{j}(x, w, b)) \quad (2.3) $$

### 2.2.5 Backpropagation

After measuring the performance of a neuron through the loss, our aim is to adjust its parameters to reduce this loss. The process of *updating* the weights to achieve this reduction is termed *backpropagation*. A common optimization technique to achieve this is *gradient descent*. For illustrative purposes, let's assume that the neuron has just one weight, and thus, there's just one weight influencing the loss (as depicted in Figure 2.3). The key insight is that moving against the slope of the loss curve brings us closer to its minimum. This slope is the derivative or the *gradient* of the loss function. Through several iterative steps, updating the weight each time, we eventually reach the ideal weight that minimizes the loss.

<footnote n="1">¹Deducing the formula for Cross-Entropy is outside the scope of this work</footnote>
<figure><img src="image_11.png" /><figcaption>FIGURE 2.3: Gradient descent illustration</figcaption></figure>

Having identified the direction of adjustment, it's also crucial to regulate the magnitude of this adjustment. This is achieved using the learning rate $\eta$, which moderates the gradient's impact on weight updates. That is, our weight updating process is calculated in the following way:

$$w_{new} = w_{old} - \eta \nabla loss(x, w_{old}) \quad (2.4)$$

### 2.2.6 Training

Up to this point, we've discussed using just one feature vector as input for a neuron. But for a comprehensive representation of our task, multiple examples are essential. This means, for effective training, we require multiple data points, or in other words, a *Dataset* comprising both features and their corresponding observed target values $(x, z_{real})$. If our *Dataset* $D$ contains $N$ items, our goal becomes minimizing the *average* loss across the dataset:

$$Loss(w, b) = \frac{1}{N} \sum_{x \in D} loss(x, w, b) \quad (2.5)$$

The entire process of executing the forward pass, determining the loss, and updating weights through backpropagation is termed an *epoch* (as illustrated in Figure 2.4).

In most cases, our computing resources might not possess the capacity to process the entire *Dataset* in one run. A common workaround is segmenting the *Dataset* into equally sized *batches*. The forward pass and loss computation are then executed batch by batch until the entire dataset is covered. Subsequently, the losses from each batch are combined to determine the final loss, followed by backpropagation. Typical batch sizes include powers of 2: 2, 4, 8, 16, 32, 64, ..., $2^n$, ...

Often, a single epoch doesn't suffice to yield satisfactory outcomes. Hence, it's a standard practice to train a neuron over multiple epochs. With each epoch, the loss typically reduces (as depicted in Figure 2.5).

### 2.2.7 Overfitting

As we progressively reduce our loss, we encounter a challenge. There are instances when the neuron will produce excellent predictions for the data it was trained on,
<figure><img src="image_12.png" /><figcaption>FIGURE 2.4: Neuron Training Process</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>FIGURE 2.5: Evolution of loss during training</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>FIGURE 2.6: Evolution of loss on the training and development splits</figcaption></figure>

but performs poorly on entirely new data, rendering the neuron useless for real-world applications. This issue is termed *overfitting*.

A prevalent strategy to counteract overfitting is to *divide* the dataset into three distinct portions:

1. The *training* split, which constitutes a portion of the dataset dedicated to the core training processes, including the forward pass, loss evaluation, and crucially, backpropagation.

2. The *development* split, a portion of the dataset designated for tracking potential overfitting. For this split, we execute the forward pass and compute the loss, and occasionally other metrics, but refrain from backpropagation. Recognizing the beginning of overfitting allows us to adjust our setup for the optimal number of epochs.

3. The *test* split, a portion of the dataset reserved for assessing the efficacy of the chosen training setup, especially since this subset wasn't involved in its selection.

When monitoring the trajectory of the loss across the training and development splits, a pattern emerges. After a specific number of epochs, the loss on the training split continues to drop, but the development split's loss starts to climb. This shift marks the beginning of overfitting (depicted in Figure 2.6).

Other techniques for avoiding overfitting, like cross-validation, neuron dropout, gradient clipping, etc., are beyond the scope of this work.

### 2.2.8 Optimizers

We may generalize Gradient Descent (Equation 2.4), by noticing that we only need to calculate some way to update our weights:

$$w_{new} = w_{old} - \text{Update}(x, w_{old}) \qquad (2.6)$$

Different optimization methods calculate the *Update* in different ways. Software implementations of these optimization methods are called *optimizers*. Some common methods are listed in Table 2.2, listed by increasing performance.
<table><caption>TABLE 2.2: Common optimization methods for backpropagation</caption><thead><tr><td>Optimization method</td><td>Description</td></tr></thead><tbody><tr><td>Gradient Descent</td><td>Classical Gradient Descent</td></tr><tr><td>Stochastic Gradient Descent (SGD)</td><td>Estimates the gradient by only using a randomly selected subset of the data</td></tr><tr><td>SGD with Momentum</td><td>Uses the estimated gradient and the previous value of Update</td></tr><tr><td>Averaged SGD</td><td>Takes into account several past values of Update and averages them while giving less importance to older values</td></tr><tr><td>Adaptive Gradient (Adagrad)</td><td>A version of SGD where each weight is updated with its own learning rate, instead of all weights using the same one. It calculates Update by averaging and normalizing over the history of updates</td></tr><tr><td>Adam</td><td>AdaGrad + Momentum</td></tr><tr><td>AdamW</td><td>Extension of Adam, controlling for the magnitude of each weight</td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>FIGURE 2.7: A Feed-Forward Neural Network</figcaption></figure>

### 2.2.9 Feed-Forward Neural Networks

As we have seen, an individual neuron may take several inputs to give an output, and this provides us with a model useful for some applications. One of the groundbreaking insights in ML was the idea of linking neurons together. This involves using the output of one neuron as the input for others. By organizing neurons into *linear layers* and interconnecting them, we create a Feed-Forward Neural Network (FFNN) (Refer to Figure 2.7)<fnref n="2" />.

When performing the forward pass, input features are fed into the FFNN via the *input layer*, they traverse through intermediate layers until they reach the *output layer*. As the FFNN undergoes training, every intermediate layer crafts new features for the subsequent layers. This mechanism enables the FFNN to *learn* new representations for the data. These in-between, or *hidden*, features detect and leverage patterns from earlier features to determine the final output of the Neural Network. A Neural Network with just one hidden layer is termed *shallow*, whereas those with several hidden layers are called *deep*.

<footnote n="2">²All diagrams of Neural Network were made with the NN - SVG tool by LeNail (2019)</footnote>
<figure><img src="image_16.png" /><figcaption>FIGURE 2.8: Neural Networks for Classification Tasks</figcaption></figure>

For classification tasks, the *SoftMax* activation function is typically employed in the output layer since it yields a probability distribution across various categories. Such a layer is termed a *SoftMax layer* or a *classification layer*. The inputs to this layer are often referred to as *logits*. Given that we're working with probabilities, the objective is to minimize the cross-entropy loss between the predictions of the Neural Network and the true labels. (Refer to Figure 2.8)

### 2.2.10 Hyperparameters

When training a Neural Network, the goal is to progressively adapt its parameters or weights to minimize the loss. However, several critical choices need to be made by the designers of the Neural Network.

Some decisions concern the *architecture* of the Neural Network, such as:

* Number of layers

* Number of Neurons per layer

* Activation functions for each layer

* Dimension of output features

* Loss function

* Optimizer

There are also decisions about specific numerical values that affect the training method, such as:

* Number of epochs

* Learning rate

* Batch size

* Sizes for the dataset splits

All these are known as *hyperparameters*. Unfortunately, there is no one-size-fits-all answer to the question of which hyperparameters to use because the optimal hyperparameters can vary depending on the specific dataset, model architecture, and task at hand. The search space for hyperparameters can be vast, and finding the best combination can be computationally expensive and time-consuming. As a result, the choice of hyperparameters is often determined through trial and error, experimentation, and a combination of domain knowledge and experience.

### 2.2.11 Neural Networks for NLP tasks

For Neural Networks to handle NLP tasks, textual data must be transformed into numerical features to feed to the network. A significant advancement in NLP came when experts discovered that instead of handcrafting algorithms for feature generation, they could train Neural Networks to produce these features for subsequent Neural Networks. The next section elaborates on this approach.
<figure><img src="image_17.png" /><figcaption>FIGURE 2.9: Naive tokenization</figcaption></figure>

## 2.3 Tokenization and Embeddings

Most NLP tasks for text start with a corpus of documents. The documents may have been manually collected by humans; automatically collected, for example, using web scrapping tools, generating text from templates, or, most recently, sampling outputs from AI tools like GPT-3 (Wang et al., 2021) and ChatGPT (Huang, Kwak, and An, 2023); or constructed using a mixture of human and automatic input.

Considering the ever-changing nature of natural language, the initial phase in Text Classification involves defining the system's Vocabulary. Tokenization refers to the act of segmenting text into smaller units, known as *tokens*. As a first approach, one may use the intuitive idea of *word* for tokens, in languages with a morphosyntax similar to English (Figure 2.9). The collection of unique tokens derived from a text corpus forms the Vocabulary.

However, naive word tokenization<fnref n="3" /> can lead to challenges with out-of-vocabulary (OOV) words when presented with new data. A more modern tokenization approach involves using *subword units*. This strategy addresses OOV words by breaking words down into smaller segments. In English, for instance, one might consider prefixes and suffixes as subwords (like *pre-*, *-ing*, *-ed*, and so on), enhancing the NLP system's ability to understand morphology. Nevertheless, instead of manually defining rules to divide words into subwords, there are two widely-used algorithms based on character distribution to create such tokenizers:

*   *Byte-Pair Encoding (BPE)* (Sennrich, Haddow, and Birch, 2016) builds a vocabulary by iteratively merging the most frequent pair of characters or character sequences in a given text corpus. It starts with a character-level vocabulary and merges the most frequent pair of tokens until a desired vocabulary size is reached.

*   *Word Piece (WP)* (Wu et al., 2016) begins by creating a vocabulary that consists of all the characters found in the training data. It then proceeds to learn a specific number of merge rules. Unlike BPE, Word Piece Tokenization selects symbol pairs that maximize their frequency relative to its constituent symbols, rather than choosing the most frequent symbol pair.

Typically, these algorithms result in different tokenizations for the same text, leading to different vocabularies. Figure 2.10 displays a sample output from the WP

<footnote n="3">³Another simple tokenization technique is *character-level tokenization*, where text is simply divided into individual characters. This tokenization technique falls outside the scope of this work.</footnote>
<figure><img src="image_18.png" /><figcaption>FIGURE 2.10: Different tokenization approaches: naive, Byte-Pair Encoding, and Word Piece</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>FIGURE 2.11: Word and Document embeddings</figcaption></figure>

method in Devlin et al. (2019) and the BPE method in Liu et al. (2019) for English content.

Even though there are Text Classification approaches which only consider the distribution of tokens in a corpus, like *Naive Bayes Classification* (Manning and Schutze, 1999), recent breakthroughs in NLP have been enabled by *text embeddings*, which are vector representations of text in high-dimensional space.

Embeddings are intended to capture the semantic relationships between texts, allowing computational models to understand and manipulate textual data. Typically, embeddings serve as *input features* for textual data, which are then fed into ML algorithms (Figure 2.12).

*Word embeddings* are a type of text embedding that assign a vector representation to each token in a text. Within Text Classification, while word embeddings are prevalent, Document Classification especially benefits from *Document embeddings*. These embeddings provide a unique vector representation for an entire document, enabling a holistic understanding and categorization of texts based on their full content and context. Essentially, text embeddings ensure that texts with analogous meanings are closely aligned, while those with divergent meanings are distinctly separated (Figure 2.11).

Text embeddings have undergone significant evolution over the years, with several techniques and models contributing to their development. We present a brief overview of the key milestones in the evolution of word embeddings:

* Early approaches focused on frequency-based representations, such as Bag-of-Words (BOW) one-hot encoding<fnref n="4" /> and term frequency-inverse document frequency (TF-IDF). These methods assigned weights or binary values to words based on their occurrence in the corpus, without capturing semantics.

<footnote n="4">⁴A one-hot vector has all its components null, except one which has value one</footnote>
<figure><img src="image_20.png" /><figcaption>FIGURE 2.12: Using Embeddings to train Neural Networks for Classification</figcaption></figure>

*   Word2Vec (Mikolov, Yih, and Zweig, 2013) popularized the concept of word embeddings trained using unsupervised learning. These models utilized shallow neural networks to learn word embeddings by predicting neighboring words or contexts.

*   Recurrent Neural Network (RNN) architectures, such as GRU (Gated Recurrent Unit), LSTM (Long Short-Term Memory), and Bi-LSTM (bidirectional LSTM), provided a breakthrough by processing sentences sequentially. For example, with InferSent document embeddings (Conneau et al., 2017). However, they faced challenges with long-term dependencies and were computationally expensive for longer sentences.

*   Contextualized word embeddings introduced the idea of generating word representations that varied depending on the context in which the tokens appeared. Models like *ElMo* (Embeddings from Language Models) (Peters et al., 2018) and OpenAI's GPT (Generative Pre-trained Transformer) (Radford et al., 2018) were able to produce dynamic word embeddings.

*   Transformer models, including Google's Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), OpenAI's GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), revolutionized the field by leveraging attention mechanisms and large-scale pre-training. These models introduced contextualized word embeddings on a larger scale and achieved state-of-the-art performance across various NLP tasks.

In this work, our primary focus is on employing the BERT family of models for
Text Classification. The details of these models will be explored in the next section.

2.4 BERT Models for Text Classification

A Language Model is a Neural Network that can be used for predicting the distribution of tokens, given a corpus. The main goal of a Language Model is to capture the underlying patterns and structure of tokens in the corpus. The architecture for the BERT (Bidirectional Encoder Representations from Transformers) family of models
<figure><img src="image_21.png" /><figcaption>FIGURE 2.13: Attention Mechanism for Machine Translation, from Olah and Carter (2016)</figcaption></figure>

was introduced Devlin et al. (2019) as one capable of learning embeddings aware of the left and right context for each token<fnref n="5" />.

The BERT family of models has gained significant popularity due to its ability to efficiently generate contextualized embeddings for both individual tokens and entire documents at a low cost (Koroteev, 2021).

While the original BERT is a general-purpose model, many tasks benefit from domain-specific knowledge. The multitude of BERT models arises from the need to cater to different languages, domains, tasks, and computational constraints, combined with ongoing research efforts to improve model performance and efficiency.

Many BERT models are freely and readily accessible and set for deployment across a range of NLP tasks. Given this availability, they will serve as primary tools for this work.

### 2.4.1 Attention and Transformers

For many years, NLP methods faced challenges with understanding distant relationships within texts. For example, consider the sentence, "The cat, which was adopted from the shelter last month, loves to play.". Here, the subject (cat) and the verb (loves) are spaced apart by several words, making it tricky for systems identifying grammatical roles. Similarly, when translating from, say, Japanese to English, it's crucial to recognize that the Japanese verb typically appears at the end of the sentence.

Initially introduced in NLP for Machine Translation (Bahdanau, Cho, and Bengio, 2015), the *Attention* mechanism enables the Neural Network to focus on particular words or phrases crucial for grasping the context. It achieves this by employing trainable parameters to determine coefficients between 0 and 1, indicating the significance of each input token to each output token (Figure 2.13). The part of the network responsible for determining these attention coefficients is termed an *Attention Head*.

Interestingly, attention can be computed between a sentence and itself to evaluate the significance of its various components relative to one another; this process is called *Self-Attention*. The outcome of the Self-Attention mechanism for a text comprising *n* tokens is an *n* × *n* matrix *A* = (*a*<sub>ij</sub>), where *a*<sub>ij</sub> is a value between 0 and 1, denoting the influence of token *j* on token *i* (typically, *p*<sub>*ij*</sub> ≠ *p*<sub>*ji*</sub>). In the original architecture, which we employ in this work, the time and memory usage for Self-Attention grow in a quadratic manner on the text's length (*O*(*n*<sup>2</sup>)). More recent Self-Attention designs aim to reduce computing time Dao et al., 2022.

Attention mechanisms are a fundamental component of *Transformer* models, a type of deep learning architecture that has revolutionized NLP tasks. The original

<footnote n="5">⁵Hence the name “Bidirectional”.</footnote>
<figure><img src="image_22.png" /><figcaption>FIGURE 2.14: Classical Transformer Architecture</figcaption></figure>

Transformer model, introduced in the “Attention is All You Need” paper by Vaswani et al., 2017, uses multiple Self-Attention Heads, and is primarily used for sequence-to-sequence tasks, such as Machine Translation.

A classical Transformer architecture consists of two main neural modules, the *Encoder* and the *Decoder*. The encoder module takes embeddings as input features, computes self-attention, and produces attention-enriched embeddings. These enhanced embeddings are then channeled to the decoder, which outputs the logits of the probabilities of the next token. Notably, the decoder also incorporates its prior outputs as inputs, applies self-attention to them, and then calculates "cross" attention between these and the attention-augmented embeddings from the encoder. The design of this architecture is depicted in Figure 2.14.

After successful training, the encoder learns the optimal way to generate embeddings for each token considering all the other tokens in the input. At the same time, the decoder learns to produce tokens in a manner informed by the input and its own output. Due to the quadratic complexity of Self-Attention, Transformer architectures have a limited input size.

## 2.4.2 The training of BERT

The architecture for BERT essentially a series of encoder modules from the traditional Transformer design, layered sequentially<fnref n="6" />. Two distinct versions were introduced by the original authors for English text: a base model with 12 encoder layers and 12 attention heads, yielding 768-dimensional embeddings, and a large model with 24 encoder layers and 16 attention heads, resulting in 1024-dimensional embeddings. The basic BERT architecture employs a WP tokenizer and can handle up to 512 tokens as input.

For training BERT to generate context-aware embeddings for individual tokens as well as entire sentences, it was subjected to two specific tasks. The datasets utilized for this purpose were Google's BookCorpus and the English portion of Wikipedia.

<footnote n="6">⁶Hence the name “Encoder Representations from Transformers”.</footnote>
<figure><img src="image_23.png" /><figcaption>FIGURE 2.15: Self-Attention for MLM</figcaption></figure>

### Masked Language Modeling

To learn embeddings for tokens, the authors introduce the Masked Language Modelling (MLM) task. At its core, the MLM task aims for the model to “fill-in the blank” or to “guess the missing word”. For MLM, once a text is tokenized, 15% of the tokens are randomly substituted with a unique [MASK] token, symbolizing a blank space. This modified sequence is then processed by the encoder layers, relying on the self-attention mechanisms to provide sufficient contextual understanding (as depicted in Figure 2.15). To determine the omitted word, the final embedding corresponding to the [MASK] token is passed through a *SoftMax* activation function, yielding a probability distribution over all the tokens in the vocabulary. By contrasting these probabilities with the known token, the cross-entropy loss is computed, which serves as the loss to be minimized. This procedure enables BERT to learn about the token distribution.

### Next Sentence Prediction

To learn embeddings for sentences, the Next Sentence Prediction task used by the authors involves joining two sentences or phrases and training the model to determine if they appeared consecutively in the training dataset. Specifically, for each example with sentences *A* and *B*, there's a 50% chance that *B* is the actual subsequent sentence to *A*, and a 50% chance it's a random sentence from the dataset. In order to do this, the authors introduce two special tokens, the [CLS] token, for classification, and the [SEP] token to separate the texts.

During training, the [CLS] token is placed at the start of the text, and the [SEP] token is positioned between the two sentences. After processing by the encoders, the [CLS] token's embedding undergoes further refinement via a *Pooling Layer*<fnref n="7" />, resulting in what's termed the aggregated or *pooled* output. This pooled output aims to encapsulate the entire text's meaning. Ultimately, this output is fed into a classification layer, and the generated probabilities are matched against the known sentence distribution to compute the cross-entropy (as illustrated in Figure 2.4.2). Through this process, BERT is trained to generate embeddings that represent whole sentences.

### 2.4.3 Using BERT for Text Classification

The training process for BERT doesn't require the texts to be labeled or categorized, making it an *unsupervised* approach. This approach allowed BERT to learn general text representations that encompass a variety of syntactic and semantic patterns across diverse contexts.

<footnote n="7">⁷This layer comprises a linear layer with a tanh activation function</footnote>
<figure><img src="image_24.png" /><figcaption>FIGURE 2.16: Next Sentence Prediction Task for BERT</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>FIGURE 2.17: Using BERT Embeddings for Document Classification</figcaption></figure>

Once the BERT model is trained, the conventional method for Document Classification using BERT involves feeding the text to the model$^a$. Then, the pooled output serves as the document's embedding. This embedding is then fed to a classification layer, which is further trained for the specific task at hand (as illustrated in Figure 2.17).

Finally, the parameters of BERT models may be adapted to new data and tasks. In Chapter 6, we detail the techniques used for fitting BERT to our task.

## 2.5 Evaluation Metrics for Classification

After training a classifier for a Classification task, including Text Classification, the subsequent phase is to assess its performance. This involves leveraging a dataset with

<footnote>$^a$keeping the second sentence empty after the [SEP] token</footnote>
predefined class labels and contrasting the classifier's predicted with these true labels.

Appendix A contains the motivations and definitions for five classic metrics for evaluating binary classifiers: accuracy, precision, recall, $F_1$, and AUC. In this section, we take a step further and discuss the $F_β$ score. The $F_β$ score extends the concept of $F_1$ by introducing a parameter $β$, which allows us to control the trade-off between precision and recall. A higher value of beta emphasizes recall, while a lower value emphasizes precision. The $F_1$ score is a special case of the $F_β$ score when $β$ is set to 1.

$$F_{\beta} = (1 + \beta^2) \frac{P * R}{\beta^2 * P + R} \qquad (2.7)$$

Example: If $β$ is set to 2, the $F_2$ score will give more two times more weight to recall than precision, making it suitable for tasks where recall is more important than precision, such as ours.

Our task involves distinguishing between events representing health risks and impacting agriculture (the positive class, “Relevant”) from harmless events (the negative class, “Irrelevant”). In this context, it becomes vital to identify as many true positive cases as possible, as they represent actual risks with potential consequences for agriculture.

Once our system flags a potential risk, subject experts will be notified to make the final assessment. A low precision would indicate that only a small fraction of the documents presented to the experts are actually relevant. Conversely, a low recall would imply that many actually relevant documents would be overlooked by the system, requiring experts to reexamine and assess the original data, a costly and time consuming task.

As a result, our focus leans more towards improving recall over precision. Therefore, we choose to assess our system using the $F_2$ score.
<figure><img src="image_26.png" /></figure>
# Chapter 3

# Related Work

As in almost all industries, Neural Networks-based AI is being adopted in Agriculture (Javaid et al., 2023). Some of the most prominent uses of AI and Deep Learning have been the automatization of decisions or alerts produced by sensors (Wongchai et al., 2022); and the use of the Internet of Things<fnref n="1" /> to integrate sensors and software, in order to warn farmers of possible issues with their crops (Bu and Wang, 2019).

On the other hand, the application of NLP to Agriculture, and especially to Epidemiological Surveillance, has been rather timid. A popular approach has been to use data mining from social media to gather textual data (Turenne et al., 2015). Twitter has been used as a source of crowdsourced knowledge about ecological events (Aramaki, Maskawa, and Morita, 2011; Charles-Smith et al., 2015). However, there have been concerns about the reliability of social media data for Epidemiological Surveillance, as the public and the agricultural workers may lack the specialized knowledge to accurately assess risks (Welvaert et al., 2017).

Moreover, until recently, the advancements in Neural Network-based Language Models didn't significantly influence NLP applications in Epidemiological Surveillance (Alrowili and Shanker, 2021; Hakala and Pyysalo, 2019). This landscape shifted during the COVID-19 Pandemic when both computer vision and NLP emerged as vital tools for diagnosis, prevention, and notably for our work, monitoring the spread of the disease. Still, these technological strides haven't been extensively applied in the field of Plant Health Surveillance.

Shankar, Bitter, and Liwicki (2020) introduced the idea of using social media data to monitor crop health. They collected a corpus of 5530 Tweets and used Binary Text Classification to detect “agriculturally relevant tweets”, using embeddings and a Support Vector Machine Classifier, and obtained an accuracy of 86.5%. Even though the authors used established embedding techniques, namely Word2Vec and Doc2Vec, nowadays, far more powerful Neural Networks exist. Based on their provided con-fusion matrix, additional metrics for their results can be deduced as: Precision at 72.4%, Recall at 21.1%, $F_1$ at 32.7%, and $F_2$ at 24.6%. The stark contrast between their reported accuracy and the other metrics likely stems from the skewed nature of their dataset, which consists of 24% relevant tweets and 76% that are not. Our research offers a marked improvement over these findings.

To the best of our knowledge, the research most aligned with ours is by Jiang et al. (2022), who introduce *ChouBERT*, a BERT model tailored for Plant Health Surveillance. *ChouBERT* emerges from adapting the French BERT model, *CamemBERT* (Martin et al., 2020), to fit a custom Plant Health Dataset. This dataset comprises Tweets and French *French Plant Health Bulletins*². Notably, the authors

<footnote n="1">¹https://en.wikipedia.org/wiki/Internet_of_things
²https://agriculture.gouv.fr/bulletins-de-sante-du-vegetal</footnote>
only disclosed the Precision metric for *ChouBERT*, which stands at 88.7%. How-
ever, since *ChouBERT* is designed exclusively for French content, its applicability to
International Plant Health Monitoring remains limited.
# Chapter 4

# The VSI Dataset

Within this Chapter, we provide a detailed description of the VSI Dataset. As mentioned in Chapter 1.3, the PESV Platform collects information on risks to Plant Health or events that could potentially affect agriculture. The VSI experts achieve this by surveying online documents, which they subsequently screen and collect into the VSI dataset.

## 4.1 Dataset Collection

<figure><img src="image_27.png" /><figcaption>FIGURE 4.1: VSI Data Collection Pipeline</figcaption></figure>

The main tool for collecting documents is the web scrapping tool *ScaleSERP*<fnref n="1" />. Every week, several queries are sent to various services, including the Google Search Engine, using the keywords and key phrases related to pathogens of interest (Table B.1).

As part of the VSI pipeline, shown in Figure 4.1, the websites corresponding to the URLs obtained from *ScaleSERP* are downloaded using a Python script. For each website, its source (in HTML) is downloaded, and the tool Trafilatura (Barbaresi, 2021) parses it to XML/TEI, while attempting to discard “non-contents”, such as menus, headings, ads, etc. Then, Trafilatura takes the XML/TEI and attempts to extract a title and an abstract from the metadata² and the full text from the XML body.

Given that these articles are extracted from public websites, they remain in their original language. The Google Translate API is used to assist the annotators with foreign-language articles. This API also automatically detects the document's language. Due to budget constraints, only the Title is translated into English.

Since documents are added each week, the VSI database is in constant construction. For our work, we utilized with the documents collected from the 11th of July

<footnote n="1">¹https://www.scaleserp.com/
²The title and the abstract are extracted from the *metadata.title* and *metadata description* fields, respectively.</footnote>
<table><caption>TABLE 4.1: Sources of text content in the VSI Dataset</caption><thead><tr><th>Content Source</th><th>Tool</th></tr></thead><tbody><tr><td>Title</td><td>Trafilatura</td></tr><tr><td>Abstract</td><td>Trafilatura</td></tr><tr><td>Full text</td><td>Trafilatura</td></tr><tr><td>Translated Title</td><td>Google Translate</td></tr></tbody></table>

2022 to the 16th of April 2023, that is, the data collected over 39 weeks, comprising approximately 35,000 documents, which corresponds to more than 800 documents per week. We focus on a subset of the VSI database, specifically its textual data, to obtain the VSI dataset (Table 4.1).

## 4.2 Dataset Annotation

There are four VSI experts dedicated to annotating the dataset. Regrettably, while constructing the dataset, there was no record maintained to track the identity of the annotator responsible for labelling each document. For this reason, we are not able to provide any inter-annotator agreement measures on this dataset (Artstein and Poesio, 2008).

For annotation, the results of the VSI pipeline mentioned above are shared with the VSI experts. They utilize the interface shown in Figure 4.2 to inspect the results for various fields, including the *Date de publication* (publication date), *Auteurs* (authors), *Organism nuisible* (pathogen), etc. Additionally, they manually input information in other fields such as *Sujet* (subject), *Fiabilité* (reliability), and come up with a *Titre* (title). It is important to note that the title written by the VSI experts is usually different from the one extracted by Trafilatura (for example, the wording may vary).

After inspection, we found that that annotators exclusively provide titles for articles they find relevant, that is, there are no titles for irrelevant articles. Since our objective is to automate the process of identifying relevant articles, we disregard the titles provided by the annotators. Instead, we rely solely on the titles generated by Trafilatura and Google Translate, which are stored even in the case of the documents being rejected.

Special attention must be given to the *Sujet* (subject), as this field will be crucial for preprocessing the dataset (see Chapter 5.3).

When the VSI experts encounter an event that catches their attention (that is, that they consider **relevant**), they assign a **subject** to the respective document and utilize the interface presented in Figure 4.3 to allocate a subject ID and description to the document. Each subject corresponds to a specific health risk event that was reported during the weeks before the experts reviewed the documents. It is possible for multiple documents to share the same subject. In the case of documents considered **irrelevant** to the monitoring of plant health, the subject field for the document is left empty.
Subsequently, after a document is deemed relevant by the VSI experts, it is offi-
cially published in the bulletin of the PESV Platform<fnref n="3" /> and can be accessed through
the VSI Document Search Engine⁴. Some sample entries containing all sources of
content can be found in Table 4.2.

<footnote n="3">³https://plateforme-esv.fr/bulletins_et_points_sur_VSI
⁴https://plateforme-esv.fr/moteur-de-recherche-vsi</footnote>
<table><caption>TABLE 4.2: Sample entries from the VSI dataset</caption><thead><tr><td>Entry ID</td><td>Title</td><td>Translated Title</td><td>Abstract</td><td>Full text</td><td>Subject ID</td></tr></thead><tbody><tr><td>68</td><td>Xylella, da giugno 47 nuovi casi</td><td>Xylella, 47 new cases since June</td><td>In Puglia la Xylella fa...</td><td>Xylella, da giugno 47 nuovi casi In Puglia...</td><td>4279</td></tr><tr><td>305</td><td>Agro - EL HERALDO - Edizione digitale</td><td>Agro - EL HERALDO - Digital edition</td><td>EL HERALDO - Edizione digital...</td><td>El presidente de la Federación del Citrus...</td><td>None</td></tr><tr><td>320</td><td>Misure fitosanitarie di controllo della Popillia japonica - 2022</td><td>Phytosanitary measures to control Popillia japonica - 2022</td><td>Il Servizio Fitosanitario Regionale per...</td><td>Popillia japonica è un insetto originario del...</td><td>4286</td></tr><tr><td>177</td><td>Traps set to catch invasive, destructive fruit fly in Pinellas County</td><td>Traps set to catch invasive, destructive fruit fly in Pinellas County</td><td>Florida Agriculture leaders are on...</td><td>PINELLAS COUNTY, Florida - Florida Agriculture leaders...</td><td>None</td></tr><tr><td>159</td><td>Preocupan los efectos del cambio climático en la variedad picual</td><td>The effects of climate change on the picual variety are of concern</td><td>Podría ver reducido su rendimiento...</td><td>1. ¿Qué son las cookies? La Web...</td><td>None</td></tr><tr><td>89</td><td>Frantoi di Puglia danneggiati dalla Xylella, in arrivo 35 milioni di euro</td><td>Puglia oil mills damaged by Xylella, 35 million euros on the way</td><td>I frantoi oleari della Puglia...</td><td>I frantoi oleari della Puglia potranno presto...</td><td>None</td></tr><tr><td>82</td><td>Dal Mipaaf un miliardo per i terreni colpiti da Xylella</td><td>From Mipaaf one billion for the land affected by Xylella</td><td>Dal Mipaaf un miliardo per...</td><td>Il V bando per i Contratti di...</td><td>None</td></tr><tr><td>61</td><td>Le ministère veut alerter sur les espèces invasives</td><td>The ministry wants to warn about invasive species</td><td>Le ministère de l'Agriculture lance...</td><td>« Plantes en danger » Le ministère...</td><td>None</td></tr><tr><td>78</td><td>Xylella, casi di Polignano portano più a nord il limite della Puglia</td><td>Xylella, cases of Polignano bring the limit of Puglia further north</td><td>Il piano di Emergenza evidenzia...</td><td>In Puglia e Basilicata SOS AGRICOLTURA Resta...</td><td>4279</td></tr></tbody></table>
<figure><img src="image_28.png" /><figcaption>FIGURE 4.2: VSI Annotation Interface for Articles</figcaption></figure>
<figure><img src="image_29.png" /><figcaption>FIGURE 4.3: VSI Annotation Interface for Subjects</figcaption></figure>
<figure><img src="image_30.png" /><figcaption>FIGURE 4.4: Distribution of subjects in the VSI dataset without pre-processing</figcaption></figure>

## 4.3 Dataset Stastistics

There are 34,587 entries in the dataset, out of which 3,715 have an assigned subject and 30,872 do not, producing a highly unbalanced dataset (Figure 4.4).

The language automatically detected by the Google Translate API on the Title can be used to approximately study the distribution of the entries in the dataset. In Figure 4.5 it can be seen that around one third of the entries are in English, followed by Italian, Spanish, and French. There is a considerable amount of failed translation attempts, for around one fifth of the entries. The complete data on documents per language is available in Appendix B.3.

The presence of some Latin documents is surprising. Upon inspecting the data, we discovered that this occurrence was due to scrapped Titles with several scientific names (Table B.2 in the Appendix).

Often, the Trafilatura scrapper is not able to retrieve content for the Title, Abstract, and Full text of a website, or the Google Translate API fails. Thus, there are several entries without content for all fields. If we check the unique entries for each source of content, we can see that there are numerous duplicate entries, since there are at least around 20,000 unique entries per source of content (Figure 4.6).

As the Translated Title is derived from the Translated Title, and due to the possibility of failure in the Google Translate API, the number of unique entries for the Translated Title is lower than that for the Title. At the same time, the Abstract field has fewer unique entries than all the other Content Sources, while the Full text has the most. This may be explained by the fact that, usually, the creators of a website omit filling the metadata, and when they do, they frequently include only a title and not a description, while most times, the body of the website will be available.

Using naive tokenization (splitting on whitespace), we can study the distribution of the tokens for all sources of content. In the histograms in Figure 4.7 the longest documents have been grouped together to facilitate plotting<fnref n="5" />. As one can see, the Title and Translated Title have a very similar distribution, both are shorter than the Abstract, while the content from the Full text is the longest.

<footnote n="5">⁵This explains the long bars at the end of the right tails of the histograms.</footnote>
<figure><img src="image_31.png" /><figcaption>FIGURE 4.5: Language distribution of entries in the VSI dataset</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>FIGURE 4.6: Unique entries per source of content in the VSI dataset</figcaption></figure>
<figure><img src="image_33.png" /><figcaption>FIGURE 4.7: Length (in n. of tokens) of the Content Sources in the VSI dataset</figcaption></figure>

## 4.4 Dataset Issues

Upon examining the data, we encountered numerous issues that we proceed to describe and which will be addressed during preprocessing (Chapter 5).

### 4.4.1 Duplicate Entries

As mentioned in Chapter 4.3, there are some duplicate entries in the dataset (Figure 4.4 VS Figure 4.6). According to the VSI experts, often the Google Search for a query will return the same results from one week to the next, and this introduces duplicate entries.

### 4.4.2 Scrapping Failures

On some instances, the Trafilatura scrapper either fails or is blocked by website servers that do not allow bots. This introduces error messages to the dataset, some samples of which can be seen in Table 4.3.
<table><caption>TABLE 4.3: Some error messages in the VSI dataset</caption><thead><tr><th>Error Message</th></tr></thead><tbody><tr><td>Not Found</td></tr><tr><td>404</td></tr><tr><td>Page Not Found</td></tr><tr><td>Your data. Your experience.</td></tr><tr><td>Loading...</td></tr><tr><td>None</td></tr><tr><td>nan</td></tr><tr><td>JavaScript n'est pas disponible.</td></tr><tr><td>JavaScript is not available.</td></tr><tr><td>Please update your browser</td></tr><tr><td>Do you accept cookies ?</td></tr><tr><td>Verify you are not a robot</td></tr><tr><td>Discuz! Database Error</td></tr></tbody></table>

### 4.4.3 Scrapping errors

In some other instances, the web scraper successfully extracts content from the website, but the parsing algorithm performs poorly, retrieving only the headers or the website's name (Tables 4.4 and 4.5).

<table><caption>TABLE 4.4: Some headers parsed by Trafilatura</caption><thead><tr><th>Header</th></tr></thead><tbody><tr><td>Search results</td></tr><tr><td>Welcome</td></tr><tr><td>Advance articles</td></tr><tr><td>shop</td></tr><tr><td>Green Blog</td></tr><tr><td>Pesquisa (<em>Query</em>)</td></tr><tr><td>Link Diretti (<em>Direct Links</em>)</td></tr><tr><td>Búsqueda (<em>Query</em>)</td></tr><tr><td>Actualidad (<em>Current events</em>)</td></tr><tr><td>Browsing</td></tr><tr><td>Project Listing</td></tr><tr><td>Research articles</td></tr><tr><td>Pagamenti Online (<em>Online Payments</em>)</td></tr><tr><td>Notícias (<em>News</em>)</td></tr><tr><td>Category: events</td></tr><tr><td>News</td></tr></tbody></table>

<table><caption>TABLE 4.5: Some website names parsed by Trafilatura</caption><thead><tr><th>Website name</th></tr></thead><tbody><tr><td>Wall Street Journal</td></tr><tr><td>Bloomberg</td></tr><tr><td>TikTok</td></tr><tr><td>Facebook</td></tr><tr><td>Portal Embrapa</td></tr><tr><td>Argentina.gob.ar</td></tr><tr><td>Notizie dal Comune (<em>Town News</em>)</td></tr><tr><td>zmianynaziemi.pl</td></tr><tr><td>Pakistan Journal of Zoology</td></tr><tr><td>SAARC Journal of Agriculture</td></tr><tr><td>MONTSAME News Agency</td></tr><tr><td>ORCID</td></tr></tbody></table>

This type of scrapping error introduces inconsistencies into the VSI dataset, as multiple documents end up having the same text for several Content Sources, as in Table 4.6⁶.

<footnote n="6">⁶X-MOL is a Chinese search engine for scholars.</footnote>
<table><caption>TABLE 4.6: Inconsistencies introduced by scrapping errors</caption><thead><tr><th>Entry ID</th><th>Title</th><th>Translated Title</th><th>Abstract</th><th>Full text</th><th>Subject</th></tr></thead><tbody><tr><td>12895</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>13860</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>4827</td></tr><tr><td>14614</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>4874</td></tr><tr><td>15033</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>4896</td></tr><tr><td>15271</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>15683</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>4930</td></tr><tr><td>16355</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>4984</td></tr><tr><td>16472</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>19065</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>19541</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>21913</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5186</td></tr><tr><td>22152</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>23491</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>23745</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>25632</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>26284</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5448</td></tr><tr><td>29990</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5694</td></tr><tr><td>30419</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5686</td></tr><tr><td>30456</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>30994</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5743</td></tr><tr><td>31122</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>31543</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>32019</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5757</td></tr><tr><td>32096</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5799</td></tr><tr><td>32279</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>32495</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr><tr><td>33089</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>5829</td></tr><tr><td>34431</td><td>X-MOL</td><td>X-MOL</td><td>Web site created using create-react-app</td><td>X-MOLYou need to enable JavaScript to run this app.</td><td>None</td></tr></tbody></table>
#### 4.4.4 Data Noise

Even in those cases when Trafilatura is able to retrieve content from the website, there are some expected types of noise in the data, considering it comes from the internet. Namely, the following (Table 4.7):

* URLs

* HTML Tags

* Emojis

* Encoding errors

* Generic noise (different quotation or division characters, strings containing only dates, etc...)

Furthermore, there exists a source of noise that may not be immediately apparent: occasionally, the content concludes with the website's name, leading to different entries that vary by only a few tokens as a suffix (Table 4.8).

#### 4.4.5 Annotation Inconsistencies

Finally, human error also introduces inconsistencies into the VSI dataset. In some cases, there are entries with the exact same content, with different or no assigned subjects (Table 4.9). After consulting with the VSI experts, they explained that this is due to two factors:

* Different annotators labeling the same content.

* After a subject loses relevance (see Chapter 4.1), the annotators usually ignore articles related to it, which is effectively equivalent to assigning no subject.

Due to all the issues mentioned, one must take the statistics in Chapter 4.3 with a grain of salt. In particular, naively checking the entries for the presence of an assigned subject (as in Figure 4.4) may lead to very similar or exactly equal content labelled in two different categories, thus rendering the classification task significantly more challenging. These issues will addressed during preprocessing (Chapter 5).
## URLs

Regione attiva piano anti Popillia japonica - www.lombardianotizie.online - Conflom-bardia

SciELO Argentina - www.scielo.org.ar

https://twitter.com/julia_lopez9/status/1601582135923863553/photo/1

Ornitho.it homepage - www.ornitho.it

Dzienna porcja warzyw - Nasza Rola - www.naszarola.pl

## HTML Tags

<strong>Fall armyworm in Europe: how can we use biocontrol? </strong>

<strong>Xylella, CIA Puglia: </strong>«<strong>Presidente Emiliano se ci sei batti un colpo</strong>»

Japankäfer <i>Popillia japonica</i>

Categoría: <span>Revista</span>

Corteva Agriscience introduces seed treatment product Dermacor<sup>®</sup> in Brazil

Current Research on <em>Fusarium</em>

[Other] Highly Sensitive and Rapid Detection of Citrus Huanglongbing Pathogen (<i>Candidatus</i> Liberibacter asiaticus') Using Cas12a-Based Methods

## Emojis

Esther Ogunbayo on LinkedIn: https://lnkd.in/dEwvQD_A Adeoye Opeyemi kindly like, follow and repost 😊

[ ⭐ ] on TikTok

#Leonardo #italy #puglia #taranto #volgopuglia #volgoitalia | Lion sculpture, Sculpture, Taranto

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

Gen L de resistencia al Virus Rugoso del Tomate (ToBRFV)

<figure><img src="image_36.png" /></figure>

Agrocultivos TV innovando

## Encoding Errors

<°°Ç¥3> ¹@Àå(Á|6Á¶°ü·Ã).

âç½ç°câé³éé³éââ ¨å½å´æâ é²æ²»ä¸è½âä¸æä°äºâ

æµè¥¿ä¹¡ææºåå¤æ ç®¡æ¤ è¶ åç¾æ£µç¾å¹´å¤æ âèææä¾â

çå±±äº°é¿åºâç§æç¿¼â æ¹åå²³é³¿âç°cèçç « âå®æ¤âæ£®æç»¿â

Î¤Î¿ ÎÎÎ©Î¤ÎÎ ÎÎÎ®ÎÎ · Î Î´Î-Î½ÎµÎ¹ ÎÎÎÎ¿Î¹ Î±Î½ÎÎ¹Î¼ÎµÎÎÎÎ¹ÎÎ · Î Î¿Î Î´Î-ÎºÎ¿Î ÎÎÎ±

å®cæ · åè¡¨æç®

<caption>TABLE 4.7: Noise in the VSI dataset</caption>
<table><caption>TABLE 4.8: Similar entries with website names as a suffix</caption><thead><tr><th>Title</th></tr></thead><tbody><tr><td>Cousin of crop-killing bacteria mutating rapidly</td></tr><tr><td>Cousin of crop-killing bacteria mutating rapidly - MixPoint</td></tr><tr><td>Cousin of crop-killing bacteria mutating rapidly - My Droll</td></tr><tr><td>Cousin of crop-killing bacteria mutating rapidly - Newsprepare</td></tr><tr><td>Cousin of crop-killing bacteria mutating rapidly - Sky News: The Latest News from the World</td></tr><tr><td>Cousin of crop-killing bacteria mutating rapidly - timetotimes.com</td></tr><tr><td>A look at 2023's new garden plants</td></tr><tr><td>A look at 2023's new garden plants | Times News Online</td></tr><tr><td>Modeling climate change impacts on potential global distribution of Tamarixia radiata</td></tr><tr><td>Modeling climate change impacts on potential global distribution of Tamarixia radiata - PubMed</td></tr><tr><td>Portugal detecta Xylella en cítricos por primera vez en la UE y en Italia se consolida el foco de 'Mosca oriental'</td></tr><tr><td>Portugal detecta Xylella en cítricos por primera vez en la UE y en Italia se consolida el foco de 'Mosca oriental' - Agrodigital</td></tr><tr><td>Portugal detecta Xylella en 75 especies vegetales</td></tr><tr><td>Portugal detecta Xylella en 75 especies vegetales - FruitToday</td></tr><tr><td>Tornano le Giornate Fai di Primavera: 750 luoghi aperti in tutta Italia</td></tr><tr><td>Tornano le Giornate Fai di Primavera: 750 luoghi aperti in tutta Italia - LegnanoNews</td></tr><tr><td>Giornate Fai di Primavera. I segreti dell'edizione 2023. Da Bolzano alla Sardegna, 750 luoghi da scoprire</td></tr><tr><td>Giornate Fai di Primavera. I segreti dell'edizione 2023. Da Bolzano alla Sardegna, 750 luoghi da scoprire - Notizie italiane in tempo reale!</td></tr><tr><td>Milleproroghe, soddisfazione di Confagricoltura per gli emendamenti a favore del settore primario - Agricolae</td></tr><tr><td>Milleproroghe, soddisfazione di Confagricoltura per gli emendamenti a favore del settore primario - Comunicati | Confagricoltura</td></tr></tbody></table>
<table><thead><tr><th>Entry ID</th><th>Title</th><th>Subject</th></tr></thead><tbody><tr><td>4662</td><td>Cousin of crop-killing bacteria mutating rapidly</td><td>None</td></tr><tr><td>5885</td><td>Cousin of crop-killing bacteria mutating rapidly</td><td>4472</td></tr><tr><td>58</td><td>Danger pour les végétaux : première détection de la bactérie Xylella fastidiosa dans le Gard</td><td>None</td></tr><tr><td>850</td><td>Danger pour les végétaux : première détection de la bactérie Xylella fastidiosa dans le Gard</td><td>4259</td></tr><tr><td>26873</td><td>Commodity risk assessment of ash logs from the US treated with sulfuryl fluoride to prevent the entry of the emerald ash borer Agrilus planipennis</td><td>5524</td></tr><tr><td>27196</td><td>Commodity risk assessment of ash logs from the US treated with sulfuryl fluoride to prevent the entry of the emerald ash borer Agrilus planipennis</td><td>5530</td></tr><tr><td>322</td><td>Anche a Varese l'invasione della Popillia Japonica, l'insetto devastatore di campi e giardini</td><td>None</td></tr><tr><td>335</td><td>Anche a Varese l'invasione della Popillia Japonica, l'insetto devastatore di campi e giardini</td><td>4286</td></tr><tr><td>343</td><td>Anche a Varese l'invasione della Popillia Japonica, l'insetto devastatore di campi e giardini</td><td>4286</td></tr></tbody></table>

<table><thead><tr><th>Entry ID</th><th>Abstract</th><th>Subject</th></tr></thead><tbody><tr><td>13997</td><td>A CABI-led study involving 57 scientists from 46 different institutions ...</td><td>4829</td></tr><tr><td>24902</td><td>A CABI-led study involving 57 scientists from 46 different institutions ...</td><td>None</td></tr><tr><td>4662</td><td>A bacterial species closely related to deadly citrus greening disease ...</td><td>4472</td></tr><tr><td>5388</td><td>A bacterial species closely related to deadly citrus greening disease ...</td><td>None</td></tr><tr><td>22658</td><td>Ministry of Agriculture activated batteries in the fight against xylella....</td><td>5245</td></tr><tr><td>24067</td><td>Ministry of Agriculture activated batteries in the fight against xylella....</td><td>5331</td></tr></tbody></table>

<table><caption>TABLE 4.9: Annotation inconsistencies in the VSI dataset</caption><thead><tr><th>Entry ID</th><th>Full text</th><th>Subject</th></tr></thead><tbody><tr><td>32437</td><td>Altre 23 piante infette dal batterio Xylella fastidiosa subsp. pauca ceppo ST53 sono state individuate tra Fasano (Brindisi) e Castellana Grotte (Bari), ...</td><td>5781</td></tr><tr><td>32712</td><td>Altre 23 piante infette dal batterio Xylella fastidiosa subsp. pauca ceppo ST53 sono state individuate tra Fasano (Brindisi) e Castellana Grotte (Bari),...</td><td>5820</td></tr></tbody></table>
<figure><img src="image_37.png" /></figure>
# Chapter 5

## Preprocessing the VSI dataset

In this Chapter, we outline the approaches taken to resolve the challenges found in the VSI Dataset. After reviewing the problems described in Chapter 4.4, we adopted various tactics to achieve a preprocessed dataset suitable for Text Classification.

### 5.1 Leveraging Keywords

Leveraging the available keywords and key phrases used for the Search Engine queries used to construct the dataset (Table B.1), we obtained four more sources of content.

Using the Python package NLTK (Bird and Loper, 2004), we divided each document into phrases and selected those containing at least one of the keywords or key phrases. This process was carried out for both the Abstract and Full text sources of content (Figure 5.1). However, considering the diverse range of languages in the documents, it was not always apparent whether we should discard the content entirely or retain it. For example, even if the keyword is not present in the text, maybe its translation is; or we could be facing a language written in a non-Latin script. Consequently, we pursued two different approaches: in one case, we discarded the original content (O.C.), and in the other, we retained it, resulting in four new sources of content (Table 5.1).

Figure 5.2 offers a comparison of the number of unique entries per content source. In both cases, keeping only the phrases containing keywords dramatically reduces the size of the data, especially for the Abstract. Possible reasons for this will be explained in Chapter 5.2. Note that, in the case of keeping the original content, there are slightly fewer unique entries for both the Abstract and the Full text (19,939 - 19,798 = 141 and 26,346 - 24,447 = 1,899, respectively). This may be explained by the fact that

<figure><img src="image_38.png" /><figcaption>FIGURE 5.1: Extracting phrases with Keywords</figcaption></figure>
<table><caption>TABLE 5.1: Sources of text content in the VSI Dataset after preprocessing</caption><thead><tr><th>Content Source</th><th>Origin</th></tr></thead><tbody><tr><td>Title</td><td>Trafilatura</td></tr><tr><td>Abstract</td><td>Trafilatura</td></tr><tr><td>Full text</td><td>Trafilatura</td></tr><tr><td>Translated Title</td><td>Google Translate</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>Preprocessing</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>Preprocessing</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>Preprocessing</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>Preprocessing</td></tr></tbody></table>

<figure><img src="image_39.png" /><figcaption>FIGURE 5.2: Unique entries per source of content in the VSI dataset</figcaption></figure>

some data noise may be removed by the sentence extraction process, especially those cases where several entries differ by a small amount of tokens (see Table 4.8).

## 5.2 Data Cleaning

### 5.2.1 Noise Removal

To address the text noise mentioned in Chapter 4.4.4, we use regular expressions to:

1. Remove characters not in any human alphabet. This handles encoding errors.

2. Remove emojis, URLs, HTML tags, hours, dates, extra white spaces.

3. Remove punctuation at beginning and end of the string.

4. Remove digits at beginning and end of the string.

5. Standardize quotation characters into the ASCII simple quotation character (').

6. Standardize hyphen-like characters into the ASCII hyphen (-).

7. Remove sentence suffixes that may be the name of the website, by deleting short text (≤ 3 tokens) that follow an ASCII hyphen or bar (|) near the end of the string.

See Table C.2 in the Appendix for some examples of the regular expressions used to clean the multilingual strings. Figure 5.3 shows the effect of string cleaning on an example text.
<figure><img src="image_40.png" /><figcaption>FIGURE 5.3: Removing noise from multilingual strings</figcaption></figure>

<table><caption>TABLE 5.2: Error Message Count per Content Source in the VSI dataset</caption><thead><tr><td>Content Source</td><td>Error Count</td><td>Percentage of Raw dataset</td></tr></thead><tbody><tr><td>Title</td><td>7224</td><td>20.89%</td></tr><tr><td>Abstract</td><td>12624</td><td>36.50%</td></tr><tr><td>Full text</td><td>6609</td><td>19.11%</td></tr><tr><td>Translated Title</td><td>7232</td><td>20.91%</td></tr><tr><td>All Content Sources</td><td>6399</td><td>18.50%</td></tr></tbody></table>

### 5.2.2 Deleting Error Messages

Fortunately, error messages in the dataset are very consistent. Given that they do not provide information on the relevance of a document, we may delete them using regular expressions. See Table C.1 in the Appendix for some examples of the regular expressions used to remove the error messages.

In Table 5.2, we present the number of error messages identified using our regular expressions. Notably, the proportion of error messages generally hovers around one-fifth of all entries. In 18.50% (≈ 20%) of cases Trafilatura completely fails to obtain any content from the website. This proportion almost doubles (36.50 ≈ 40%) for the Abstract. This discrepancy in the Abstract could be attributed to the common occurrence of missing metadata, as discussed in Chapter 4.3.

### 5.2.3 Handling scraping errors

In order to handle scraping errors from Trafilatura, such as obtaining the name of the website or certain HTML headers, our first approach was to use regular expressions to filter them out. After removing the error messages from all sources of content, we proceeded to count the occurrences of each entry and ranked them in descending order. Given that our dataset contains approximately 35,000 documents, we focused on entries that appeared at least twice.

Although we attempted to identify common patterns and create regular expressions to match frequent entries, this approach proved unsuccessful. It only filtered around 500 unique entries, accounting for a mere 2% of all unique entries in the best case (Abstract). Consequently, it became apparent that several scraping errors likely appeared only once in the dataset, rendering our previous strategy ineffective.

Nevertheless, along this examination, we noticed that scraping errors tend to be short in length (see Tables 4.4 and 4.5). As a result, we devised a new filtering strategy based on the length of the entries. However, simply counting naive tokens
<table><caption>TABLE 5.3: Short Message Count per Content Source in the VSI dataset</caption><thead><tr><th>Content Source</th><th>Short Message Count</th><th>Percentage of Raw dataset</th></tr></thead><tbody><tr><td>Title</td><td>5691</td><td>16.45%</td></tr><tr><td>Abstract</td><td>1023</td><td>2.96%</td></tr><tr><td>Full text</td><td>181</td><td>0.52%</td></tr><tr><td>Translated Title</td><td>5306</td><td>15.34%</td></tr><tr><td>All Content Sources</td><td>17</td><td>0.05%</td></tr><tr><td>Title & Abstract</td><td>589</td><td>1.70%</td></tr><tr><td>Title & Full text</td><td>141</td><td>0.41%</td></tr><tr><td>Full text & Abstract</td><td>50</td><td>0.06%</td></tr></tbody></table>

was insufficient, as languages like Chinese often lack whitespace in their text. Therefore, we needed a more robust algorithm to determine when a string is considered suspiciously “too short.”

Our heuristic to filter by length consists in:

1. First, the string is split on white spaces.

2. Then, if there is only one token, and the string has less than 20 characters, it is considered “too short”.

3. Next, if there are less than four tokens, the string is considered “too short”

4. Finally, in all other cases, we keep the string.

Upon removing error messages, we observed that parsing errors occur in entries for the Title and the Translated Title approximately 1 in 7 times (≈ 16%), as can be seen in Table 5.3. The Abstract and Full text are rarely short (≤ 3%), and very few entries have more than one content source that is too brief (≤ 3%).

## 5.3 Resolving Inconsistencies and Duplicate Documents

The initial preprocessing steps focused solely on cleaning the textual content of the dataset. Now, our attention turns to addressing the labeling challenges, specifically, instances where entries have identical content but differ in subject assignment or lack a subject (as in Tables 4.6 and 4.9).

Considering our main goal is to automate the identification of articles relevant to the VSI experts, where subject assignment signifies their interest, we propose the following strategy:

> For each of the eight content sources, we group the dataset based on the text content. If a particular text has been assigned a subject at least once, we categorize it as **relevant**. Conversely, if no subject has been assigned to a text, we consider it **irrelevant**.

We believe this strategy reflects the intention of the annotators as they labeled the documents (see Chapter 4.2). Additionally, by design, this strategy eliminates duplicate content. Table 5.4 illustrates this process for the content from the Title.
<table><thead><tr><th>Entry ID</th><th>Title</th><th>Subject</th></tr></thead><tbody><tr><td>4662</td><td>Cousin of crop-killing bacteria mutating rapidly</td><td>None</td></tr><tr><td>5885</td><td>Cousin of crop-killing bacteria mutating rapidly</td><td>4472</td></tr><tr><td>19661</td><td>Modeling climate change impacts on potential ...</td><td>None</td></tr><tr><td>19361</td><td>Modeling climate change impacts on potential ...- PubMed</td><td>None</td></tr><tr><td>58</td><td>Danger pour les végétaux : première détection de ...</td><td>None</td></tr><tr><td>850</td><td>Danger pour les végétaux : première détection de ...</td><td>4259</td></tr><tr><td>26873</td><td>Commodity risk assessment of ash logs from the US ...</td><td>5524</td></tr><tr><td>27196</td><td>Commodity risk assessment of ash logs from the US ...</td><td>5530</td></tr><tr><td>29896</td><td>Tornano le Giornate Fai di Primavera ...</td><td>None</td></tr><tr><td>29899</td><td>Tornano le Giornate Fai di Primavera ...- LegnanoNews</td><td>None</td></tr><tr><td>...</td><td>...</td><td>...</td></tr></tbody></table>

<figure><img src="image_41.png" /></figure>

<table><caption>TABLE 5.4: Determining Relevance for Title</caption><thead><tr><th>Title</th><th>Relevance</th></tr></thead><tbody><tr><td>Cousin of crop-killing bacteria mutating rapidly</td><td>1</td></tr><tr><td>Modeling climate change impacts on potential ...</td><td>0</td></tr><tr><td>Danger pour les végétaux : première détection de ...</td><td>1</td></tr><tr><td>Commodity risk assessment of ash logs from the US ...</td><td>1</td></tr><tr><td>Tornano le Giornate Fai di Primavera ...</td><td>0</td></tr><tr><td>...</td><td>...</td></tr></tbody></table>
<figure><img src="image_42.png" /><figcaption>FIGURE 5.4: Unique Entries per content source before and after preprocessing</figcaption></figure>

## 5.4 Preprocessing Results

In this section, we present the outcomes of our preprocessing techniques, yielding refined datasets suitable for text classification. Additional numerical results can be found in Appendix D.1.

Figure 5.4 demonstrates that preprocessing leads to a reduction in the number of unique entries. For the majority of Content Sources, approximately 5 out of 6 unique entries are retained after the filtering process (≈ 85%), except for the Title and Abstract, where approximately 4 out of 6 unique entries are preserved (≈ 67%).

<table><caption>TABLE 5.5: Unique Entries per content source before and after preprocessing</caption><thead><tr><th>Unique entries per Content Source</th><th>Percentage left</th></tr></thead><tbody><tr><td>Title</td><td>67.49%</td></tr><tr><td>Abstract</td><td>85.13%</td></tr><tr><td>Full text</td><td>85.53%</td></tr><tr><td>Translated Title</td><td>68.65%</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>85.95%</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>85.02%</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>87.17%</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>85.41%</td></tr></tbody></table>

Regarding the average token count of the content, the histograms in Figure 5.5 show that distributions across the four original Content Sources align with the original observations in Figure 4.7. The same trend is observed for the Phrases with Keywords as well. Upon comparing the means before and after preprocessing (Table 5.6), we observe that the means are higher for all the original content sources, except for the Full text.

The higher means can be attributed to the filtering of error messages, which tend to be short in length, and short entries that may correspond to scrapping errors. As for the Full text, upon closer inspection of the filtered-out *long* texts identified as error messages, it was observed that they often pertain to topics such as website cookies or terms and conditions. These texts tend to be quite lengthy and are not typically found in website metadata (from where the Title and the Abstract are extracted).
This observation explains the lower mean for the Full text after preprocessing, which we interpret as a positive indication of our filtering approach.

When analyzing the Phrases with Keywords, a significant decrease in the mean is noted, particularly for the Phrases with Keywords extracted from the Full text. This reduction can be attributed to the fact that only a few sentences in an article usually contain the exact keywords used for locating them through a Search Engine. Consequently, after preprocessing, the mean of the Phrases with Keywords decreases substantially when compared to their original sources.

Overall, these observations highlight the effectiveness of our preprocessing in removing undesirable content while retaining relevant information.

<table><caption>TABLE 5.6: Mean token counts before and after preprocessing the VSI dataset</caption><thead><tr><th>Content Source</th><th>Before preprocessing</th><th>After preprocessing</th></tr></thead><tbody><tr><td>Title</td><td>11.29</td><td>12.46</td></tr><tr><td>Abstract</td><td>50.02</td><td>50.55</td></tr><tr><td>Full text</td><td>1102.29</td><td>1097.76</td></tr><tr><td>Translated Title</td><td>11.66</td><td>12.82</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>(50.02)</td><td>34.92</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>(50.02)</td><td>35.77</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>(1102.29)</td><td>115.66</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>(1102.29)</td><td>217.71</td></tr></tbody></table>

By applying our preprocessing approach, we construct eight labelled datasets for binary Text Classification (Figure 5.6). These datasets exhibit a balance of positives and negatives ranging from 12 to 14% positives, and from 86 to 88% negativesª, representing a slight improvement compared to the original dataset, which had a balance of 10% positives to 90% negatives (Figure 4.4).

The only exception are the Phrases with Keywords from the Abstract discarding Original Content (“Phrases with Keywords (Abstract)”). This specific dataset displays a 27% positive and 73% negative balance. The reasons behind this imbalance can be attributed to the factors previously explained. As the metadata required for extracting the Abstract may be empty or uninformative; and few entries may contain the exact search keywords we are looking for; the dataset is substantially reduced, resulting in only 4,050 unique entries for the Phrases with Keywords (Abstract) as opposed to the original 19,939 unique entries for the Abstract (≈ 20% left). Consequently, the significantly smaller dataset size has led to a noticeable change in the balance of the labels.

<footnote>ªFor more details, see Table D.3 in Appendix D</footnote>
<figure><img src="image_43.png" /><figcaption>(a) Title</figcaption></figure>

<figure><img src="image_44.png" /><figcaption>(b) Abstract</figcaption></figure>

<figure><img src="image_45.png" /><figcaption>(c) Full text</figcaption></figure>

<figure><img src="image_46.png" /><figcaption>(d) Translated Title</figcaption></figure>

<figure><img src="image_47.png" /><figcaption>(e) Phrases with Keywords (Abstract)</figcaption></figure>

<figure><img src="image_48.png" /><figcaption>(f) Phrases with Keywords + O.C. (Abstract)</figcaption></figure>

<figure><img src="image_49.png" /><figcaption>(g) Phrases with Keywords (Full text)</figcaption></figure>

<figure><img src="image_50.png" /><figcaption>(h) Phrases with Keywords + O.C. (Full text)</figcaption></figure>

<caption>FIGURE 5.5: Length (in n. of tokens) of the Content Sources in the VSI dataset after preprocessing</caption>
<figure><img src="image_51.png" /><figcaption>(a) Title</figcaption></figure>

<figure><img src="image_52.png" /><figcaption>(b) Abstract</figcaption></figure>

<figure><img src="image_53.png" /><figcaption>(c) Full text</figcaption></figure>

<figure><img src="image_54.png" /><figcaption>(d) Translated Title</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>(e) Phrases with Keywords (Abstract)</figcaption></figure>

<figure><img src="image_56.png" /><figcaption>(f) Phrases with Keywords + O.C (Abstract)</figcaption></figure>

<figure><img src="image_57.png" /><figcaption>(g) Phrases with Keywords (Full text)</figcaption></figure>

<figure><img src="image_58.png" /><figcaption>(h) Phrases with Keywords + O.C (Full text)</figcaption></figure>

<caption>FIGURE 5.6: Positive/Negative balance after preprocessing the VSI dataset</caption>
<figure><img src="image_59.png" /></figure>
# Chapter 6

# Methodology

In this Chapter, we delve into the techniques we used to train BERT for Document Classification. We first introduce different pre-trained BERT models, and then proceed to explain the training approaches used for this work.

## 6.1 BERT Models

As previously mentioned in Chapter 2.4, the BERT architecture from Devlin et al. (2019) has been customized for various tasks, resulting in a wide array of models to select from. For our project, we have opted to employ six different BERT models, based on their specialized task and the characteristics of our data. Since our data is in different languages, we utilize BERT models that differentiate between upper case and lower case characters (cased versions). Additionally, due to computational constraints, we opt for the basic (base) versions of the models rather than the large ones (Table 6.1).

### 6.1.1 BERT-base

The original BERT model developed by Google in Devlin et al. (2019). It was trained on a large corpus of English text from Wikipedia and Google's BooksCorpus dataset, using a WP tokenizer.

### 6.1.2 mBERT

The Multilingual BERT (Devlin et al., 2019) is a version of the original BERT model that is designed to handle text in multiple languages. It was trained on the top 104 languages with the biggest Wikipedia sizes by number of articles, using a WP tokenizer.

### 6.1.3 Bio-Link-BERT

This model was introduced in Yasunaga, Leskovec, and Liang (2022) as the result of training a classical BERT architecture while enriching the context of the documents by

<table><caption>TABLE 6.1: Sizes of BERT models used for this project. Adapted from Conneau et al. (2020)</caption><thead><tr><th>Model</th><th>Languages</th><th>Tokenization</th><th>Encoder Layers</th><th>Attention Heads</th><th>Vocabulary</th><th>Parameters</th></tr></thead><tbody><tr><td>BERT-base</td><td>1</td><td>WordPiece</td><td>12</td><td>12</td><td>30k</td><td>110M</td></tr><tr><td>mBERT</td><td>104</td><td>WordPiece</td><td>12</td><td>12</td><td>110k</td><td>172M</td></tr><tr><td>Bio-Link-BERT</td><td>1</td><td>WordPiece</td><td>12</td><td>12</td><td>30k</td><td>110M</td></tr><tr><td>SciBERT</td><td>1</td><td>WordPiece</td><td>12</td><td>12</td><td>30k</td><td>110M</td></tr><tr><td>RoBERTa-base</td><td>1</td><td>BPE</td><td>12</td><td>8</td><td>50k</td><td>125M</td></tr><tr><td>XLM-RoBERTa</td><td>100</td><td>SentencePiece</td><td>12</td><td>12</td><td>250k</td><td>270M</td></tr></tbody></table>
using two corpora of documents linked to one another by hyperlinks. For documents *doc*₁, *doc*₂ linked by a hyperlink *doc*₁ → *doc*₂, during the training phase, the sections of *doc*₂ that are reachable by the hyperlink will be concatenated to their corresponding sections from *doc*₁ for the Next Sentence Prediction task (See Chapter 2.4.2). This process increases the context available for the model to learn embeddings.

The authors present a model for the general domain (Link-BERT), trained on the English Wikipedia with hyperlinks; and a model for the biomedical domain (Bio-Link-BERT), trained on PubMed<fnref n="1" /> abstracts (in English) containing hyperlinks, and used a WP tokenizer. As of the culmination of our work, Bio-Link-BERT is still one of the top performing BERT models in the Biomedical Language Understanding and Reasoning Benchmark (BLURB) (Gu et al., 2021).

### 6.1.4 SciBERT

This model was introduced in Beltagy, Lo, and Cohan (2019) as a specialized model for scientific texts. It was trained with the classical BERT architecture using 1.14 million papers in English from Semantic Scholar², 82% of which belong to the biomedical domain. The authors use the full text from the papers.

### 6.1.5 RoBERTa-base

This model was introduced in Liu et al. (2019) and is an extension and improvement of the original BERT model. RoBERTa-base follows the same architecture as BERT but undergoes a more extensive pre-training process, specifically, it uses much larger batch sizes, longer sequences, and removes the Next Sentence Prediction” task used in BERT. Additionally, it is trained on 10 times more data than BERT, by merging five datasets: Google's BooksCorpus, the English Wikipedia, Common-Crawl³ English News (Hamborg et al., 2017), OpenWebText (Gokaslan et al., 2019), and the “Stories” subset of the Common-Crawl (Trinh and Le, 2018). It uses a BPE tokenizer.

### 6.1.6 XLM-RoBERTa

This model was introduced in Conneau et al. (2020) as the multilingual version of RoBERTa-base. For training, the authors constructed the 100-CC corpus for training XLM-RoBERTa, which consists of monolingual data in 100 languages from the Common-Crawl, and used a SentencePiece tokenizer (Kudo and Richardson, 2018), which extends the classical BPE tokenization algorithm.

### 6.1.7 Excluded Models

Initially, we considered three other BERT models; however, after careful consideration, we decided not to include them due to various reasons.

**ChouBERT** This model was introduced in Jiang et al. (2022) as a specialized model for Plant Health Monitoring. However, it was trained exclusively on French texts, which raises strong concerns about its ability to generalize to our linguistically and thematically diverse dataset.

<footnote n="1">¹https://pubmed.ncbi.nlm.nih.gov/
²https://www.semanticscholar.org/
³https://commoncrawl.org/</footnote>
<figure><img src="image_60.png" /><figcaption>FIGURE 6.1: Fine-tuning BERT for Document Classification</figcaption></figure>

**DistilBERT** This model was introduced in Sanh et al. (2019) as a smaller version of the original BERT model. It uses a process called “distillation”, which compresses the knowledge of a larger BERT model into a smaller one, while still retaining decent performance.

During our preliminary training experiments with the BERT models mentioned above, DistilBERT consistently underperformed. Regardless of the configurations used, it defaulted to predicting the majority class. Therefore, we chose to omit DistilBERT from our final model selection.

**BioBERT** This model was introduced in Lee et al. (2020) as a BERT variant specifically designed for biomedical text and tasks. Once again, this model uses the classical BERT architecture on PubMed abstracts in English. Since it has been vastly outperformed by Bio-Link-BERT in several NLP tasks (Yasunaga, Leskovec, and Liang, 2022), we decided not to include it in our final model selection.

## 6.2 Fine-tuning

Finetuning, in the context of NLP, refers to the process of taking a pre-trained language model and further training it on a specific task or dataset. At the outset, we use a pre-trained model, which has undergone training on a vast dataset, enabling it to grasp general features and patterns applicable to a broad spectrum of tasks. Fine-tuning allows us to adapt the pre-trained model to a more specific domain by updating its parameters using task-specific data.

In our scenario, BERT embeddings are used for document classification. To tailor the model to our particular task, we enable the backpropagation process to adjust not only the classification layer but also the weights of the BERT model (refer to Figure 6.1 in contrast to Figure 2.17). This technique allows us to make the BERT model adapt the way it produces embeddings and learn a representation of our documents which is more convenient for our task.

Given the imbalance in our datasets, we employ weighted cross-entropy as the loss. Yet, our initial experiments revealed that artificially balancing the dataset enhanced performance. Further details can be found in Chapter 6.4.
<figure><img src="image_61.png" /><figcaption>FIGURE 6.2: Intuition behind PET</figcaption></figure>

## 6.3 Pattern-Exploiting Training

Few-shot learning is a ML set of techniques that aim to train models to generalize and perform well on tasks with very limited labeled data. Pattern-Exploiting Training (PET) was introduced by Schick and Schütze (2021) as a technique for Few-shot learning in NLP using basic prompt engineering, and has provided improvements over standard training approaches.

The intuition behind PET is to use *task descriptions* to improve the model's performance in Text Classification. Leveraging BERT's MLM feature, we design a *pattern*, which takes a document and produces a text with one concealed token (using the [MASK] token). The model then tries to “fill-in the blank” by estimating the likelihood of all tokens. The pattern is intended to guide the model towards specific predictions. Next, we use a *verbalizer*, a list of words tied to a category. We then combine these predictions to determine the final likelihood for each category, as illustrated in Figure 6.2. Finally, we aggregate the probabilities for the relevant tokens into artificial or *synthetic* scores for each category (Figure 6.2).

Implementing PET is more intricate. It requires creating multiple patterns, with each pattern aimed at guiding the model's token predictions by emphasizing a different aspect of the task. Additionally, each category requires its own verbalizer. It's important to remember that the model's vocabulary is tied to its tokenization method. For instance, in Figure 6.2, while we use terms like “Definitely”, in reality it might be tokenized differently, like “Definite-ly”. We're generally limited to concealing a single token, even though there are ways to expand this¹.

In our case, we aim to determine if a document is relevant to Plant Health Surveillance using the BERT models discussed in Chapter 6.1. Each BERT model has a unique vocabulary and tokenization technique, so we had to pick words that would be consistently tokenized across models. As our task is Binary Classification, we opted to create patterns using polar questions⁵. Given the 512-token input limit of

<footnote n="4">⁴The PET paper's authors suggest a method for multiple tokens, however, the trade-off is that the training time increases significantly

⁵Polar questions typically have a “yes/no” answer</footnote>
<table><caption>TABLE 6.2: Patterns and Verbalizers used for PET</caption><thead><tr><th>Pattern</th><th>Verbalizers</th></tr></thead><tbody><tr><td>Is this about plant health? [____] &lt; document &gt;</td><td>Relevant → yes</td></tr><tr><td>Is this relevant? [____] &lt; document &gt;</td><td>Irrelevant → no</td></tr><tr><td>Is this about epidemic surveillance? [____] &lt; document &gt;</td><td></td></tr></tbody></table>

BERT models and the length of some documents, we ensured the masked token was positioned early in the input, avoiding patterns like the following:

$$ \begin{gathered} \langle document \rangle \langle question \rangle [____] \\ \langle document \rangle [____] \langle question \rangle \end{gathered} $$

After initial tests revealed that PET training took much longer than Fine-tuning, we settled on just three patterns. The specific patterns and verbalizers we used are detailed in Table 6.2.

The PET method involves two steps: first training multiple models to generate synthetic labels, followed by fine-tuning a model for classification.

Unlike conventional training approaches, PET requires dividing the dataset into four distinct splits: training, development, testing, and an additional split of unlabeled documents known as the *unlabeled* split. The rationale behind this will become clear as we delve deeper into the method. Given that this technique is designed for Few-shot learning, the training split typically has only few entries per category.

The first phase involves training a set, or *ensemble*, of models. For every pattern, we take a copy of a BERT model, such as mBERT, and prompt it to predict the concealed token for a specific document in the training segment. Each ensemble model yields different probability estimates for the tokens in the vocabulary.

These probabilities are then combined by averaging their logits for each token<fnref n="6" /> and using *SoftMax* to determine a probability for every vocabulary token. Subsequently, the verbalizers transform the probabilities of pertinent tokens into category label probabilities. These synthetic label probabilities are then compared against the true labels in order to calculate a cross entropy loss. This procedure is depicted in Figure 6.3⁷.

Simultaneously, during this phase, the ensemble models undergo training for MLM on our dataset to learn the typical contexts for our task. This method of training an ensemble for both synthetic annotation and MLM on a dataset requires a custom loss function, which the authors define as a weighted sum of the cross-entropy loss and the MLM loss.

The second phase of PET involves utilizing the trained PET ensemble to generate synthetic labels for the unlabeled documents. This synthetically labelled dataset is then employed to fine-tune a new copy of the BERT model (mBERT in our example) for Document Classification, as detailed in Chapter 6.2 above. The fine-tuned model, along with its trained classification layer, will act as our Document Classifier (See Figure 6.4).

It's worth noting that each phase in PET could, in principle, have distinct hyper-parameters. Yet, mirroring the original authors' approach, we opted to use the same hyperparameters across both phases for our experiments.

<footnote n="6">⁶Meaning, the aggregated logit for a token is the average of the logits generated by the ensemble models.</footnote>

<footnote n="7">⁷This approach bears some similarity to the "distillation" method introduced by Sanh et al. (2019)</footnote>
<figure><img src="image_62.png" /><figcaption>FIGURE 6.3: Training a PET Ensemble</figcaption></figure>

<figure><img src="image_63.png" /><figcaption>FIGURE 6.4: Creating a Synthetically Labeled Dataset with the PET Ensemble, and subsequently Fine-tuning</figcaption></figure>

In this work, we opted to utilize PET with varying training sizes, including 50, 100, 200, 500, and 1000 documents *per category*.

Next, we detail the method we used to divide the dataset for Fine-tuning and PET.

## 6.4 Splitting the Dataset

As mentioned at the end of Chapter 5.4, after preprocessing, we obtain eight datasets for Binary Text Classification. For each dataset, we produce seven splits, two for training using Fine-tuning and five for training using PET.

The most straightforward splitting method is for Fine-tuning on an unbalanced dataset. For each Content Source, such as the Translated Title, we randomly split the dataset into 80%-10%-10% for training, development, and testing, maintaining the original dataset's category distribution.

For Fine-tuning on a balanced dataset, we retain the split sizes and replicate the test split from the previous step. We then create balanced development and training segments by oversampling the positive category and randomly discarding from the negative category. Considering the original distribution was 90% negatives to 10% positives, we believe there will still be a sufficient representation of the negative category.
<figure><img src="image_64.png" /><figcaption>FIGURE 6.5: Strategies for Dataset Splitting</figcaption></figure>

<table><caption>TABLE 6.3: Code repositories for this work</caption><thead><tr><td>Repository</td><td>URL</td></tr></thead><tbody><tr><td>Preprocessing and Training</td><td>GitLab Repository</td></tr><tr><td>Original PET</td><td>GitHub Repository</td></tr><tr><td>PET for PESV</td><td>GitLab Repository</td></tr></tbody></table>

For PET, we select a specific number of documents per category, for instance, 50. We then replicate the test and development splits from the balanced Fine-tuning split. From the balanced Fine-tuning training split, we select the first 50 documents from both the positive and negative categories to form the PET training split (keeping in mind that PET was first proposed for Few-shot learning). The remaining documents are used to construct the unlabeled split, by discarding their labels. Likewise for 100, 200, 500, and 1000 documents per category.

A detailed breakdown of the split sizes can be found in Appendix E.2.

This approach guarantees that for every Content Source, the test split remains consistent across all training methods. Maintaining the same test split for different training techniques ensures the **comparability** of our results. This is crucial, as performance on the test split assesses the classifier's effectiveness in real-world situations.

## 6.5 Implementation, Hyperparameters, and Hardware

For this work, we worked in Python 3.9.2 and implemented a package for preprocessing the datasets and training BERT language models. This package is built around the **Transformers** library from the 😊 HuggingFace company for Fine-tuning (Wolf et al., 2020), **Scikit-learn** for metrics and statistics (Pedregosa et al., 2011), **pandas** for handling the datasets (Team, 2020), **BeautifulSoup4** for parsing XML (Richardson, 2019), among others. We also modified the code repository for PET to include our own task-specific patterns and verbalizers. Our implementation is designed to run on one GPU.

During training, we keep track of the accuracy, precision, recall, $F_1$, and $F_2$ on the training and development splits per epoch. For the final models, we calculate all these metrics along with the Area Under the Curve (AUC) on the development and test splits. Memory constraints prevented us from saving model checkpoints during
<table><caption>TABLE 6.4: Hyperparameters</caption><thead><tr><th>Hyperparameter</th><th>Value</th></tr></thead><tbody><tr><td>Input size</td><td>300 tokens</td></tr><tr><td>Batch size</td><td>32</td></tr><tr><td>Optimizer</td><td>AdamW</td></tr><tr><td>Train-dev-test splits</td><td>80%-10%-10%</td></tr><tr><td>Epochs for Fine-tuning</td><td>10</td></tr><tr><td>Learning rate for Fine-tuning</td><td>5e-5</td></tr><tr><td>Epochs for PET Ensemble</td><td>5</td></tr><tr><td>Epochs for PET Document Classifier</td><td>5</td></tr><tr><td>Learning rate for PET</td><td>1e-5</td></tr></tbody></table>

training. Therefore, we conducted two training rounds: one to determine the best number of epochs and another to train only up to those identified optimal epochs.

We performed some experiments on the MaiAGE local computers, but the bulk of the training was performed on the Lab-IA cluster<fnref n="8" />. This is a computing platform associated to the University of Paris-Saclay, and serves researchers in the Paris region. For our experiments, we conducted various tests and executed our code, but the GPU selection was beyond our control, contingent on the availability of GPUs. Table E.1 shows the specifications of this cluster.

After several preliminary experiments, it became clear that the size of our datasets and the implementations of our training methods (specifically, PET) would pose a challenge for the available hardware. These preliminary tests also shed light on certain patterns in the training results, which informed our hyperparameter choices. We found that Fine-tuning was faster than PET, that larger batch sizes maxed out our GPU memory, and that extremely high or low learning rates led to erratic learning. Table 6.4 lists the hyperparameters we used for our final setup. For any hyperparameters not highlighted in this table, we relied on the default settings offered by the Transformers and the original PET implementations.

## 6.6 Methodology Summary

To summarize, we have eight Content Sources (each one has a labeled dataset), seven training setups (two for Fine-tuning and five for PET), and six different BERT models. This gives us $8 \times 7 \times 6 = 336$ training scenarios.

Table 6.5 shows the approximate training times for each Content Source for single instances of Fine-tuning and PET. Since there are two Fine-tuning setups and five PET setups, the total training time was approximately $2 \times 41 + 5 \times 102 = 594$ GPU hours, or about 3.5 GPU weeks. Factoring in the two training cycles, the total comes to about 7 GPU weeks.

<footnote n="8">⁸http://hebergement.universite-paris-saclay.fr/lab-ia/</footnote>
<table><caption>TABLE 6.5: Approximate Training times per Content Source, in GPU hours</caption><thead><tr><td>Content Source</td><td>Fine-tuning Time</td><td>PET Time</td></tr></thead><tbody><tr><td>Title</td><td>3</td><td>10</td></tr><tr><td>Abstract</td><td>4</td><td>11</td></tr><tr><td>Full text</td><td>12</td><td>24</td></tr><tr><td>Translated Title</td><td>3</td><td>10</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>1</td><td>4</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>4</td><td>11</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>3</td><td>12</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>11</td><td>20</td></tr><tr><td>Total</td><td>41</td><td>102</td></tr></tbody></table>
<figure><img src="image_65.png" /></figure>
# Chapter 7

# Results and Discussion

In this Chapter, we present the results of our training experiments, and provide insights on them.

Comprehensive training plots and logs can be found in our GitLab repository. For a glimpse into the results from our initial training round, refer to Figures 7.1 and 7.2. In the first figure, we can see that the optimal epoch to avoid overfitting in that particular scenario is the fourth epoch, which coincidentally also maximizes the $F_2$. In contrast, in the second image, the first epoch is optimal, while it does not have the best $F_2$.

<figure><img src="image_66.png" /><figcaption>FIGURE 7.1: Unbalanced Fine-tuning - Losses and Development $F_2$ evolution for mBERT on the Title</figcaption></figure>

<figure><img src="image_67.png" /><figcaption>FIGURE 7.2: PET 1000 - Losses and Development $F_2$ evolution for RoBERTa-base on the Title</figcaption></figure>

The optimal epochs for each training scenario are detailed in Appendix F.1. It's evident that Unbalanced Fine-tuning takes the longest to reach the optimal epoch, especially when compared to the other methods, where the majority reach the optimum within the first or second epoch. The disparity between the two fine-tuning techniques is significant, with Balanced Fine-tuning often reaching its peak performance in just one epoch. When it comes to PET, all five scenarios exhibit a consistent pattern, typically achieving their optimal epoch before the fourth one. This suggests
that any variations in the rate of convergence might occur during the training of the PET ensembles. Given that all ensembles undergo training for five epochs, it must be that this sufficient for achieving satisfactory results. Unfortunately, we did not record the evolution of the custom losses used to train the PET ensembles.

<figure><img src="image_68.png" /><figcaption>FIGURE 7.3: Development $F_2$ for mBERT at the optimal epochs, across all the training methods</figcaption></figure>

Based on these findings, we moved forward to the second round, trained models up to their optimal epochs, and evaluated their final performance. The metrics for the best classifiers across each training scenario and for every BERT model are detailed in Appendix F.2. A direct comparison of the Development $F_2$ outcomes reveals that Unbalanced Fine-tuning consistently underperforms compared to other training techniques <fnref n="1" /> (See Figure 7.3 for sample results for mBERT). In fact, Balanced Fine-tuning outperforms Unbalanced Fine-tuning in nearly all instances, with only one exception<fnref n="2" />.

When examining PET, a noticeable trend emerges: performance improves as the document count per category grows. Specifically, the Development $F_2$ for PET 50 generally lags behind other PET variants and seldom matches their levels. Conversely, both PET 500 and PET 1000 consistently register superior results.

Given the high number of training scenarios (totaling 336), we may impose a stringent performance threshold, focusing only on classifiers that achieve a Development $F_2$ of at least 80%. Out of all the scenarios, only 72 classifiers meet this criterion, which is roughly one-fifth of the total. A summary of these top-performing classifiers is provided in Table F.20. It's evident from the table that PET training typically surpasses Fine-tuning<fnref n="3" />, and that Unbalanced Fine-tuning never makes it to the list. Among the PET configurations, both PET 500 and PET 1000 consistently exceed the set threshold.

When ranking the BERT models based on the number of top-performing classifiers, XLM-RoBERTa leads with 22, followed closely by mBERT and Bio-Link-BERT, each with 16. This suggests that the multilingual nature of our dataset might play a more pivotal role in determining relevance than its biomedical aspect. This aligns with the fact that only 4 top classifiers are associated with SciBERT, and a mere 3 with BERT-base. RoBERTa-base, with its 11 top classifiers, occupies a middle ground, possibly indicating its superior capabilities compared to BERT-base and SciBERT. However, its training exclusively on English text might be a limiting factor.

At the same time, it's concerning to observe that several top classifiers exhibit worryingly low Development AUC values, ranging between 60% and 80% (Figure

<footnote n="1">¹A $F_2$ score of 0% indicates the model defaulted to predict the predominant class (negative)</footnote>

<footnote n="2">²This specific instance pertains to BERT-base on the Title, showing a marginal difference of about 3%</footnote>

<footnote n="3">³The sole exception being XLM-RoBERTa on the Full text</footnote>
7.4). A low AUC indicates challenges faced by these models in differentiating between positive and negative instances. Yet, their high $F_2$ scores suggest a pronounced recall relative to precision. Piecing these insights together, it's plausible that these models achieve elevated $F_2$ scores by predominantly categorizing most documents as positive.

<figure><img src="image_69.png" /><figcaption>FIGURE 7.4: Sample AUCs for some classifiers</figcaption></figure>

Prioritizing models that genuinely grasp the concept of relevance rather than indiscriminately marking everything as positive, we set a threshold of 80% for the Development AUC. By doing so, we narrow down to 49 classifiers, detailed in Table F.21. Interestingly, the Content Source with the highest number of top-performing classifiers is the Full text, exhibiting 15 top classifiers across all BERT models. This indicates that the Full text, typically longer than other Content Sources, offers ample content, enabling all models to effectively learn the task.

Conversely, it's worth noting that the Phrases with Keywords (Abstract) doesn't feature on the list. As discussed in Chapter 5, this dataset undergoes significant reduction during preprocessing, potentially leaving it with insufficient content. Given that we still have four top-performing classifiers for the Abstract, we opt to **exclude** the Phrases with Keywords (Abstract) as a Content Source.

To avoid classifiers that flag almost everything as a positive, we impose a 75% threshold on the Development $F_1$. This criterion narrows our selection to 17 classifiers. Table F.22 presents the Development and Test metrics for these classifiers, emphasizing the top-performing metrics. Once more, the Full text dominates with the most classifiers on this list (9), and its best three metrics are highlighted.

Predictably, the Test metrics are lower than the Development metrics. Yet, it's striking to see a significant drop in the $F_1$ and precision, with most falling below the 50% mark. This trend can be attributed to our emphasis on the $F_2$, which leans towards Recall at the expense of Precision. At the same time, the highest Test Recalls hover around or exceed 80%.

Observe that the Phrases with Keywords (Full text) is absent from the list. This can be attributed to the significant reduction in dataset content when only sentences containing keywords are considered. Additionally, the peak metrics for Phrases with Keywords + O.C (Abstract) and Phrases with Keywords + O.C (Full text) consistently fall short of those for Abstract and Full text, respectively. These findings suggest that incorporating Phrases with Keywords as Content Sources does **not** enhance performance, contrary to our hypothesis in Chapter 5.1. Consequently, we opt to **exclude** these sources of content moving forward, and thus, we are left with the four original Content Sources.

Considering the best performing classifiers (Table 7.1), it's noteworthy that, except for one, all top classifiers are derived from PET 1000. Moreover, the best classifiers are consistently obtained by training multilingual BERT models. This highlights
the significance of the multilingual nature of our dataset in determining relevance over its biomedical aspect.

<table><caption>TABLE 7.1: Best classifiers by Content Source</caption><thead><tr><th>Content Source<br>Model<br>Training Method</th><th>Title<br>mBERT<br>PET 1000</th><th>Abstract<br>XLM-RoBERTa<br>PET 1000</th><th>Full text<br>mBERT<br>PET 1000</th><th>Full text<br>XLM-RoBERTa<br>Balanced Fine-tuning</th><th>Full text<br>XLM-RoBERTa<br>PET 1000</th><th>Translated Title<br>mBERT<br>PET 1000</th></tr></thead><tbody><tr><td>Development F<sub>2</sub></td><td>82.44 %</td><td>82.34 %</td><td>82.75 %</td><td>83.62 %</td><td>85.73 %</td><td>82.87 %</td></tr><tr><td>Development AUC</td><td>83.46 %</td><td>84.15 %</td><td>88.82 %</td><td>90.57 %</td><td>88.85 %</td><td>85.27 %</td></tr><tr><td>Development F<sub>1</sub></td><td>77.55 %</td><td>77.41 %</td><td>81.52 %</td><td>83.95 %</td><td>81.6 %</td><td>78.81 %</td></tr><tr><td>Development Accuracy</td><td>75.09 %</td><td>74.91 %</td><td>81.06 %</td><td>84.06 %</td><td>79.99 %</td><td>76.92 %</td></tr><tr><td>Development Precision</td><td>70.58 %</td><td>70.4 %</td><td>79.56 %</td><td>84.52 %</td><td>75.53 %</td><td>72.86 %</td></tr><tr><td>Development Recall</td><td>86.06 %</td><td>85.98 %</td><td>83.58 %</td><td>83.39 %</td><td>88.73 %</td><td>85.82 %</td></tr><tr><td>Test F<sub>2</sub></td><td>60.56 %</td><td>60.57 %</td><td>66.97 %</td><td>63.72 %</td><td>63.79 %</td><td>63.81 %</td></tr><tr><td>Test AUC</td><td>82.5 %</td><td>84.6 %</td><td>89.61 %</td><td>87.22 %</td><td>89.66 %</td><td>84.2 %</td></tr><tr><td>Test Precision</td><td>27.83 %</td><td>28.74 %</td><td>50.21 %</td><td>51.1 %</td><td>44.31 %</td><td>31.86 %</td></tr><tr><td>Test Recall</td><td>85.78 %</td><td>83.76 %</td><td>79.24 %</td><td>82.24 %</td><td>72.49 %</td><td>85.15 %</td></tr><tr><td>Test F<sub>1</sub></td><td>42.03 %</td><td>42.79 %</td><td>35.44 %</td><td>38.42 %</td><td>29.4 %</td><td>46.37 %</td></tr><tr><td>Test Accuracy</td><td>66.73 %</td><td>69.14 %</td><td>86.13 %</td><td>76.28 %</td><td>90.15 %</td><td>72.9 %</td></tr></tbody></table>

In summary, among all the training methods examined in our work, Unbalanced Fine-tuning yields the least favorable performance, while PET demonstrates increasingly improved performance as more documents per category are introduced. In terms of Content Sources, incorporating keywords from the documents does not enhance performance, whereas utilizing the Full text proves to be the most effective in providing substantial content for the models to learn from, with the Title, Translated Title and Abstract lagging behind. Additionally, the multilingual aspect of our dataset appears to hold greater significance than their biomedical content, as evidenced by the superior performance of classifiers trained with multilingual BERT models.

Finally, we put forward a reason for the enhanced performance of PET compared to Fine-tuning. We theorize that the use of task descriptions in PET enables the model to have a clearer comprehension of the task. To illustrate this idea, we reference an example from the foundational PET paper by Schick and Schütze (2021).

Take into account the next three statements: the first two have labels, and our challenge is to assign a label to the third.

1. Category A: This *was* the best *pizza* I've ever had.

2. Category B: You *can* get better *sushi* for half the *price*.

3. Category ?: *Pizza was* average. Not worth the *price*.

Based solely on this data, determining a label is challenging: Is the focus on pizza or price? Or is it about identifying verb tenses (Past vs. Present)? Yet, if we're informed that the objective is to identify sentences discussing pizza, we'd promptly label the third sentence in category A. Conversely, if it's about prices, category B would be the appropriate label.

For our task, consider the three following (artificial) sentences:

1. Relevant: Oriental Fruit fly (Bactrocera dorsalis) detected in a new location in Central Asia.

2. Irrelevant: Check out the best decorative plants for your garden this summer, and beware of insects!

3. Category ?: What to do if you find fruit flies plaguing your field - Agriculture News Online.
For the task of identifying document relevance for Plant Health Surveillance, we categorize the third sentence as 'Irrelevant'. However, such subtleties might be overlooked by a language model without additional context.

Indeed, referring to Table 7.1, it's evident that Fine-tuning yields one of the top classifiers only when applied to the Full text. Throughout our analysis, we've observed that many top-performing classifiers were also trained on the Full text. This suggests that due to its length (averaging 1102 tokens, but cut at 300 input tokens), the Full text provides ample content, enabling any BERT model to grasp the task effectively. In contrast, other Content Sources, being much shorter than the Full text, might not offer sufficient context for the models. For instance, the Title averages 11 tokens, the Abstract 50 tokens, and the Translated Title 12 tokens. As a result, classifiers benefit from the added context given by PET patterns.

Furthermore, it's worth noting that all other best classifiers predominantly use PET 1000. This suggests that exposing the model to more document examples combined with task-specific knowledge enhances its performance. However, the performance improvement stales over time, as the incremental gains from transitioning from PET 500 to PET 1000 are marginal, as seen in Tables F.8-F.13.

In conclusion, by employing PET and its patterns, we inject task-specific knowledge, thereby guiding the model.
<figure><img src="image_70.png" /></figure>
# Chapter 8

# Practical Implications and
Future Work

In this Chapter, we discuss the real-world impact of our research, detailing its application, updates, and the effects it will have on the work of the VSI experts.

## 8.1 Overview

Upon determining the optimal hyperparameters for the most effective classifiers (as seen in Table 7.1), the PESV team plans to integrate our findings into their data collection pipeline as a component of the TIERS-ESV initiative.

For the TIERS-ESV project, we have developed two key services:

* A service for inference.

* A service for training.

## 8.2 Inference Service

<figure><img src="image_71.png" /><figcaption>FIGURE 8.1: Our classifiers added to the VSI Data Collection Pipeline</figcaption></figure>

Our Document Classification system will be integrated into the VSI data collection pipeline, as depicted in Figure 8.1. This system is composed of two primary components:

1. Heuristic methods

2. Neural-based Text Classification

The heuristic component is designed to eliminate entries that lack substantial content. For those documents that do contain content, the Text Classification component of the system applies our trained models. The results produced by this filter will be fed into the VSI database. Overall, we anticipate that this addition will significantly reduce the manual effort required by the VSI experts.

The implementation of this filter will utilize the Python package developed during this thesis.
1. First, the results from web scraping, as an XML/TEI file, are processed using our tool to extract the Title, the Abstract, the Full text, and the Translated Title.

2. Subsequently, any noise such as HTML tags, URLs, website names, and the like are filtered out from the text for each Content Source.

3. The heuristic module then employs the tools we developed for preprocessing. By using functions that identify error messages and scraping issues, one can eliminate document that lack substantial content.

4. Then, for each Content Source, the cleaned text is given as input for the Text Classification model.

5. Finally, the output of the filter for a particular document consists of

* A boolean flag indicating whether the document was kept.

* The cleaned text

* The prediction: Relevant VS Irrelevant

* The label probabilities

Given our observations on addressing error messages and managing scraping errors (as discussed in Chapters 5.2.2 and 5.2.3), we anticipate that the heuristic model will exclude between 20% to 30% of unique documents, depending on the Content Source.

This filtering system will be applied to the documents scraped weekly by the PESV Platform. For user convenience, it will be encapsulated within a REST API, hosted on the MaiAGE servers, and integrated into the *AlvisNLP-ML<fnref n="1" />* toolkit of the Bibliome group (Ba and Bossy, 2016). Lastly, when publishing selected documents in the official bulletin of the PESV Platform, they will be arranged in order of decreasing relevance.

## 8.3 Training Service

As the VSI Database expands with new entries, there will be a need to update the models utilized for Text Classification. This responsibility falls on the training service, which implements the entire pipeline to train the best performing classifiers from Table 7.1.

The frequency and data selection for training will be at the discretion of the VSI experts. They must determine whether training should occur weekly, bi-weekly, etc. Additionally, as their priorities and interests evolve, they might find it beneficial to limit the training data, perhaps only focusing on data from recent months.

The training service is designed to run on the Lab-IA cluster. All scripts related to this service can be found in our GitLab repository.

## 8.4 Consequences for Plant Health Surveillance

Until now, the VSI experts at the PESV Platform have been meticulously reviewing every scraped document. Integrating Text Classification into the annotation process will significantly reduce the burden on the VSI experts, ultimately aiding in long-term Plant Health Surveillance.

<footnote n="1">¹https://bibliome.github.io/alvisnlp/</footnote>
By incorporating the Heuristics module, the VSI experts can identify which queries and websites pose challenges for their scraping tools.

Given that the VSI experts will review documents marked as Relevant by our system, we decided to calculate the positive ratio for the top classifiers on their respective test splits. Moreover, we aim for a reduced rate of false negatives, meaning the percentage of genuinely Relevant documents that our system mistakenly labels as Irrelevant (refer to Table 8.1). The rationale behind this is that overlooking an event that poses a risk to Plant Health (a positive) is more dangerous than mistakenly identifying a harmless event (a negative) as Relevant.

<table><caption>TABLE 8.1: Positives and FN ratios for the best classifiers</caption><thead><tr><th>Content Source</th><th>Model</th><th>Training Method</th><th>Positives/Total</th><th>FN/Total</th></tr></thead><tbody><tr><td>Title</td><td>mBERT</td><td>PET 1000</td><td>43.33%</td><td>2.0%</td></tr><tr><td>Abstract</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>40.16%</td><td>2.23%</td></tr><tr><td>Full text</td><td>mBERT</td><td>PET 1000</td><td>26.55%</td><td>1.66%</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>Balanced Fine-tuning</td><td>24.16%</td><td>2.89%</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>37.27%</td><td>1.20%</td></tr><tr><td>Translated Title</td><td>mBERT</td><td>PET 1000</td><td>34.62%</td><td>1.69%</td></tr></tbody></table>

The minimal False Negative rates indicate that our systems will overlook only around 2 out of every 100 Relevant (unique) documents, which we view as an impressive outcome. As mentioned in Chapter 5.4, the actual proportions in the VSI Dataset fluctuate between 12% and 14% positives. The closer our classifiers approach to this balance, the fewer documents the VSI experts will need to review manually.

In the most favorable scenario, with a Positive ratio of 24%, the VSI experts will encounter roughly 76% ≈ 3/4 fewer documents than they currently do. Conversely, in the least favorable scenario, with a Positive ratio of 43%, they will see about 57% ≈ 1/2 fewer documents.

This demonstrates that integrating our system into their monitoring mission will significantly decrease the number of documents requiring manual review. This will allow the VSI experts to either assess the same volume of documents more quickly, allocate the same time to evaluate a larger set of documents, or broaden their surveillance to additional pathogens. We consider this outcome to be satisfactory and a fitting conclusion to our work.
## 8.5 Future Work

There are some ways for improving our work, here, we highlight three.

**Translating the Full text** One significant improvement could involve translating the Full text of the documents. Our findings indicate that using the Full text provides the most substantial content for training our models. Considering that the most advanced language models are typically developed initially for English text, creating a new Content Source by translating the Full text into English seems like a logical step.

However, this approach comes with its own set of challenges:

* Using the Google Translate API for extensive content could become costly.

* Determining whether a particular species is associated to an agricultural event is important to determine its relevant to Plant Health Surveillance. Given that agricultural reports tend to use the common names of species, using automatic translation carries the risk of inaccurately rendering the common names of various organisms. For instance, the Coccinellidae beetle is referred to as 'ladybug' in English, while in French, it is known as 'Bête à bon Dieu' (literally, 'god's creature'). Such discrepancies present a substantial challenge for general-purpose translation systems.

**Leveraging Metadata** As mentioned in Chapter 4, the VSI Database also contains metadata for each article, which presents an opportunity to enhance our results.

For instance, considering that the interests of the VSI experts evolve over time, we could leverage the publication and annotation dates associated with each article to craft additional features for Machine Learning. This could potentially allow the model to adapt more effectively to the shifting focus of the VSI experts.

Additionally, we could parse the URL of each article and use it to help determine which sources are the most trustworthy.

**Clustering instead of Classification** A final suggestion is to complement Text Classification with *Clustering* techniques. This approach would enable us to analyze the similarities between documents and understand their distribution more comprehensively.

For instance, we could investigate how documents are related across different subjects, or assess whether one subject is encompassed within another. This could provide valuable insights into the structure and relationships within the dataset, and to improve the efficiency of the VSI experts.
# Chapter 9

# Conclusions

This master thesis has explored an application of NLP in the field of Plant Health Surveillance, a critical area that impacts agriculture, food security, and environmental sustainability. Conducted at the INRAE Laboratory, a renowned French research institute, and in collaboration with the PESV Platform this work stands at the intersection of advanced computational methods and biological research.

The research was aimed at automating the process of determining the relevance of documents for Plant Health Surveillance, thereby significantly reducing the manual effort required by the experts of the VSI team at the PESV Platform. The thesis thoroughly studies the properties and statistics of the data on phytosanitary events collected and annotated by the VSI experts and develops methods to preprocess it for NLP applications.

Furthermore, the thesis outlines the development of a Document Classification system, which is intended to be integrated into the VSI data collection pipeline. This system comprises two primary components: Heuristic methods and Neural-based Text Classification. The heuristic component is designed to filter out entries that lack substantial content. For documents that do contain content, the Text Classification component of the system applies trained models to categorize the documents as 'Relevant' or 'Irrelevant' for Plant Health Surveillance. The results produced by this filter are intended to be fed into the VSI database.

The thesis provides a detailed analysis of the performance of various BERT models, with mBERT and XLM-RoBERTa producing the best results, due to the multilingual nature of the data. The Pattern Exploiting Training (PET) method, which was used extensively in this research, proved to be effective, especially when numerous examples (from 500 to 1,000 per category) were provided to the model during training. The thesis shows that using the document's Full text, due to its length and richness, allows the models to grasp the task effectively, highlighting the importance of context in Text Classification tasks.

As the VSI Database expands with new entries, there will be a need to update the models used for Text Classification. The thesis outlines a training service that is designed to be flexible and adaptive to the evolving needs and priorities of the VSI experts.

In summary, this thesis makes a contribution to the field of Text Classification in the context of Plant Health Surveillance. It not only provides a robust and automated solution to a real-world problem but also offers insights into the effectiveness of various training strategies and the importance of context in Text Classification tasks. The work presented in this thesis holds promise for significantly reducing the manual workload of the VSI experts, thus aiding long-term Plant Health Surveillance.
<figure><img src="image_72.png" /></figure>
Bibliography

Alrowili, Sultan and Vijay Shanker (June 2021). "BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA". In: *Proceedings of the 20th Workshop on Biomedical Language Processing*. Online: Association for Computational Linguistics, pp. 221–227. DOI: 10.18653/v1/2021.bionlp-1.24. URL: https://aclanthology.org/2021.bionlp-1.24.

Aramaki, Eiji, Sachiko Maskawa, and Mizuki Morita (July 2011). "Twitter Catches The Flu: Detecting Influenza Epidemics using Twitter". In: *Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing*. Edinburgh, Scotland, UK.: Association for Computational Linguistics, pp. 1568–1576. URL: https://aclanthology.org/D11-1145.

Artstein, Ron and Massimo Poesio (2008). "Survey Article: Inter-Coder Agreement for Computational Linguistics". In: *Computational Linguistics* 34.4, pp. 555–596. DOI: 10.1162/coli.07-034-R2. URL: https://aclanthology.org/J08-4004.

Ba, Mouhamadou and Robert Bossy (May 2016). "Interoperability of corpus processing workflow engines: the case of. AlvisNLP/ML in OpenMinTeD". In: *Meeting of working Group Medicago sativa*. Portoroz, Slovenia, np. URL: https://hal.science/hal-01455853.

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio (2015). "Neural Machine Translation by Jointly Learning to Align and Translate". In: *3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings*. Ed. by Yoshua Bengio and Yann LeCun. URL: http://arxiv.org/abs/1409.0473.

Barbaresi, Adrien (2021). "Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction". In: *Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations*. Association for Computational Linguistics, pp. 122–131. URL: https://aclanthology.org/2021.acl-demo.15.

Beltagy, Iz, Kyle Lo, and Arman Cohan (Nov. 2019). "SciBERT: A Pretrained Language Model for Scientific Text". In: *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. Hong Kong, China: Association for Computational Linguistics, pp. 3615–3620. DOI: 10.18653/v1/D19-1371. URL: https://aclanthology.org/D19-1371.

Bird, Steven and Edward Loper (July 2004). "NLTK: The Natural Language Toolkit". In: *Proceedings of the ACL Interactive Poster and Demonstration Sessions*. Barcelona, Spain: Association for Computational Linguistics, pp. 214–217. URL: https://aclanthology.org/P04-3031.

Brown, Tom et al. (2020). "Language models are few-shot learners". In: *Advances in neural information processing systems* 33, pp. 1877–1901.

Bu, Fanyu and Xin Wang (2019). "A smart agriculture IoT system based on deep reinforcement learning". In: *Future Generation Computer Systems* 99, pp. 500–507. ISSN: 0167-739X. DOI: https://doi.org/10.1016/j.future.2019.01001.
04.041. URL: https://www.sciencedirect.com/science/article/pii/S0167739X19307277.

Charles-Smith, Lauren E. et al. (2015). *Using Social Media for Actionable Disease Surveillance and Outbreak Management: A Systematic Literature Review*. en. Ed. by Lidia Adriana Braunstein. DOI: 10.1371/journal.pone.0139701. URL: http://dx.doi.org/10.1371/journal.pone.0139701.

Conneau, Alexis et al. (Sept. 2017). "Supervised Learning of Universal Sentence Representations from Natural Language Inference Data". In: *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*. Copenhagen, Denmark: Association for Computational Linguistics, pp. 670–680. DOI: 10.18653/v1/D17-1070. URL: https://aclanthology.org/D17-1070.

Conneau, Alexis et al. (July 2020). "Unsupervised Cross-lingual Representation Learning at Scale". In: *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*. Online: Association for Computational Linguistics, pp. 8440–8451. DOI: 10.18653/v1/2020.acl-main.747. URL: https://aclanthology.org/2020.acl-main.747.

Costa, Liliane Soares da, Italo L Oliveira, and Renato Fileto (2023). "Text classification using embeddings: a survey". In: *Knowledge and Information Systems*, pp. 1–43.

Dao, Tri et al. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness". In: *arXiv*. DOI: 10.48550/ARXIV.2205.14135. URL: https://arxiv.org/abs/2205.14135.

Devlin, Jacob et al. (2019). *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. arXiv: 1810.04805 [cs.CL].

Gokaslan, Aaron et al. (2019). *OpenWebText Corpus*. http://Skylion007.github.io/OpenWebTextCorpus.

Gu, Yu et al. (Oct. 2021). *Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing*. en. DOI: 10.1145/3458754. URL: http://dx.doi.org/10.1145/3458754.

Hakala, Kai and Sampo Pyysalo (Nov. 2019). "Biomedical Named Entity Recognition with Multilingual BERT". In: *Proceedings of the 5th Workshop on BioNLP Open Shared Tasks*. Hong Kong, China: Association for Computational Linguistics, pp. 56–61. DOI: 10.18653/v1/D19-5709. URL: https://aclanthology.org/D19-5709.

Hamborg, Felix et al. (Mar. 2017). "news-please: A Generic News Crawler and Extractor". In: *Proceedings of the 15th International Symposium of Information Science*. Berlin, pp. 218–223. DOI: 10.5281/zenodo.4120316.

Hanley, James A and Barbara J McNeil (1982). "The meaning and use of the area under a receiver operating characteristic (ROC) curve." In: *Radiology* 143.1, pp. 29–36.

Huang, Fan, Haewoon Kwak, and Jisun An (2023). "Is ChatGPT Better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech". In: *Companion Proceedings of the ACM Web Conference 2023*. WWW '23 Companion. Austin, TX, USA: Association for Computing Machinery, pp. 294–297. ISBN: 9781450394192. DOI: 10.1145/3543873.3587368. URL: https://doi.org/10.1145/3543873.3587368.

Javaid, Mohd et al. (2023). "Understanding the potential applications of Artificial Intelligence in Agriculture Sector". In: *Advanced Agrochem* 2.1, pp. 15–30. ISSN: 2773-2371. DOI: https://doi.org/10.1016/j.aac.2022.10.001. URL: https://www.sciencedirect.com/science/article/pii/S277323712200020X.
Jiang, Shufan et al. (2022). "ChouBERT: Pre-training French Language Model for Crowdsensing with Tweets in Phytosanitary Context". In: *International Conference on Research Challenges in Information Science*. Springer, pp. 653-661.

Koroteev, M. V. (2021). "BERT: A Review of Applications in Natural Language Processing and Understanding". In: *arXiv*. DOI: 10.48550/ARXIV.2103.11943. URL: https://arxiv.org/abs/2103.11943.

Kudo, Taku and John Richardson (Nov. 2018). "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing". In: *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*. Brussels, Belgium: Association for Computational Linguistics, pp. 66-71. DOI: 10.18653/v1/D18-2012. URL: https://aclanthology.org/D18-2012.

Lee, Jinhyuk et al. (2020). "BioBERT: a pre-trained biomedical language representation model for biomedical text mining". In: *Bioinformatics* 36.4, pp. 1234-1240.

LeNail, Alexander (2019). "NN-SVG: Publication-Ready Neural Network Architecture Schematics". In: *Journal of Open Source Software* 4.33, p. 747. DOI: 10.21105/joss.00747. URL: https://doi.org/10.21105/joss.00747.

Liu, Yinhan et al. (2019). *RoBERTa: A Robustly Optimized BERT Pretraining Approach*. cite arxiv:1907.11692. URL: http://arxiv.org/abs/1907.11692.

Manning, Christopher and Hinrich Schutze (1999). *Foundations of statistical natural language processing*. MIT press.

Martin, Louis et al. (July 2020). "CamemBERT: a Tasty French Language Model". In: *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*. Online: Association for Computational Linguistics, pp. 7203-7219. DOI: 10.18653/v1/2020.acl-main.645. URL: https://aclanthology.org/2020.acl-main.645.

Mikolov, Tomáš, Wen-tau Yih, and Geoffrey Zweig (2013). "Linguistic regularities in continuous space word representations". In: *Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies*, pp. 746-751.

Ministère de L'Agriculture et de L'Alimentation (2018). *Convention cadre portant définition et organisation de la Plateforme national d'epidémiosurveillance en santé végétale*. Convention cadre. Direction générale de l'Alimentation.

Olah, Chris and Shan Carter (2016). "Attention and Augmented Recurrent Neural Networks". In: *Distill*. DOI: 10.23915/distill.00001. URL: http://distill.pub/2016/augmented-rnns.

Pedregosa, F. et al. (2011). "Scikit-learn: Machine Learning in Python". In: *Journal of Machine Learning Research* 12, pp. 2825-2830.

Peters, Matthew E. et al. (June 2018). "Deep Contextualized Word Representations". In: *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*. New Orleans, Louisiana: Association for Computational Linguistics, pp. 2227-2237. DOI: 10.18653/v1/N18-1202. URL: https://aclanthology.org/N18-1202.

Radford, Alec et al. (2018). "Improving language understanding by generative pre-training". In: *OpenAI Blog*.

Radford, Alec et al. (2019). "Language models are unsupervised multitask learners". In: *OpenAI blog* 1.8, p. 9.

Richardson, Leonard (2019). "Beautiful Soup 4 documentation". In: *dec*.

Sanh, Victor et al. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter". In: *NeurIPS EMC2 Workshop*.
Schick, Timo and Hinrich Schütze (Apr. 2021). "Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference". In: *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*. Online: Association for Computational Linguistics, pp. 255-269. DOI: 10.18653/v1/2021.eacl-main.20. URL: https://aclanthology.org/2021.eacl-main.20.

Sennrich, Rico, Barry Haddow, and Alexandra Birch (2016). *Neural Machine Translation of Rare Words with Subword Units*. arXiv: 1508.07909 [cs.CL].

Shankar, Priyamvada, Christian Bitter, and Marcus Liwicki (2020). "Digital Crop Health Monitoring by Analyzing Social Media Streams". In: *2020 IEEE / ITU International Conference on Artificial Intelligence for Good (AI4G)*, pp. 87-94. DOI: 10.1109/AI4G50087.2020.9310985.

Shen, Dou (2009). "Text Categorization". In: *Encyclopedia of Database Systems*. Boston, MA: Springer US, pp. 3041-3044. ISBN: 978-0-387-39940-9. DOI: 10.1007/978-0-387-39940-9_414. URL: https://doi.org/10.1007/978-0-387-39940-9_414.

Steadman, R. G. (1979). "The Assessment of Sultriness. Part I: A Temperature-Humidity Index Based on Human Physiology and Clothing Science". In: *Journal of Applied Meteorology (1962-1982)* 18.7, pp. 861-873. ISSN: 00218952, 2163534X. URL: http://www.jstor.org/stable/26179216 (visited on 08/02/2023).

Team, The Pandas Development (Feb. 2020). *pandas-dev/pandas: Pandas*. https://doi.org/10.5281/zenodo.3509134. DOI: \url{10.5281/zenodo.3509134}.

Trinh, Trieu H. and Quoc V. Le (2018). "A Simple Method for Commonsense Reasoning". In: *ArXiv abs/1806.02847*. URL: https://api.semanticscholar.org/CorpusID:47015717.

Turenne, Nicolas et al. (2015). "Open Data Platform for Knowledge Access in Plant Health Domain: VESPA Mining". Other. working paper or preprint. URL: https://hal.science/hal-01145489.

Van Rijsbergen, C J (Jan. 1979). *Information Retrieval*. 2nd ed. Oxford, England: Butterworth-Heinemann.

Vaswani, Ashish et al. (2017). "Attention is All you Need". In: *Advances in Neural Information Processing Systems*. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

Wang, Shuohang et al. (Nov. 2021). "Want To Reduce Labeling Cost? GPT-3 Can Help". In: *Findings of the Association for Computational Linguistics: EMNLP 2021*. Punta Cana, Dominican Republic: Association for Computational Linguistics, pp. 4195-4205. DOI: 10.18653/v1/2021.findings-emnlp.354. URL: https://aclanthology.org/2021.findings-emnlp.354.

Welvaert, Marijke et al. (Feb. 2017). "Limits of use of social media for monitoring biosecurity events". In: *PLOS ONE* 12.2, pp. 1-17. DOI: 10.1371/journal.pone.0172457. URL: https://doi.org/10.1371/journal.pone.0172457.

Wolf, Thomas et al. (Oct. 2020). "Transformers: State-of-the-Art Natural Language Processing". In: *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*. Online: Association for Computational Linguistics, pp. 38-45. DOI: 10.18653/v1/2020.emnlp-demos.6. URL: https://aclanthology.org/2020.emnlp-demos.6.

Wolfram, Stephen (Feb. 2023). *What Is ChatGPT Doing and Why Does It Work?*. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/.
Wongchai, Anupong et al. (2022). “Artificial intelligence - enabled soft sensor and internet of things for sustainable agriculture using ensemble deep learning architecture”. In: *Computers and Electrical Engineering* 102, p. 108128. ISSN: 0045-7906.
DOI: https://doi.org/10.1016/j.compeleceng.2022.108128. URL: https://www.sciencedirect.com/science/article/pii/S0045790622003780.

Wu, Yonghui et al. (2016). *Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation*. arXiv: 1609.08144 [cs.CL].

Yasunaga, Michihiro, Jure Leskovec, and Percy Liang (2022). *LinkBERT: Pretraining Language Models with Document Links*. arXiv: 2203.15827 [cs.CL].

Zhang, Aston et al. (2023). *Dive into Deep Learning*. https://d2l.ai/. arXiv: 2106.11342 [cs.LG].
<figure><img src="image_73.png" /></figure>
# Appendix A

## Evaluation Metrics for Classification

In Binary Classification, we deal with two classes. Our primary concern is to measure how well the positive class is predicted. The content of this section is an adaptation of Van Rijsbergen (1979).

When the classifier is applied to an item, there are four potential outcomes, contingent on the classifier's predictions and the true class labels. These four cases constitute a *Confusion Matrix* (as depicted in Table A.2).

<table><caption>TABLE A.1: Classification Outcomes</caption><thead><tr><td>Case</td><td>Description</td></tr></thead><tbody><tr><td>True Positives (TP)</td><td>The item <i>is</i> in the positive class, and the classifier <i>predicts</i> the positive class.</td></tr><tr><td>True Negatives (TN)</td><td>The item <i>is</i> in the negative class, and the classifier <i>predicts</i> the negative class.</td></tr><tr><td>False Negative (FN)</td><td>The item <i>is</i> in the positive class, and the classifier <i>predicts</i> the negative class.</td></tr><tr><td>False Positive (FP)</td><td>The item <i>is</i> in the negative class, and the classifier <i>predicts</i> the positive class.</td></tr></tbody></table>

<table><caption>TABLE A.2: Confusion Matrix for Binary Classification</caption><thead><tr><td>Actual Class/Predicted Class</td><td>Predicted Positive</td><td>Predicted Negative</td></tr></thead><tbody><tr><td>Actual Positive</td><td>TP</td><td>FN</td></tr><tr><td>Actual Negative</td><td>FP</td><td>TN</td></tr></tbody></table>

There are four classic metrics for evaluating classifiers:

* **Accuracy:** Accuracy measures the overall correctness of a classification model. It is the ratio of *correctly* predicted instances to the total number of instances in the dataset.

$$A = \frac{TP + TN}{Positives + Negatives} \qquad (A.1)$$

Example: If a model classifies 90 out of 100 samples correctly, the accuracy would be 90%.

Example: In an unbalanced dataset containing 80% of negatives, a classifier that flags everything as a negative would get an accuracy of 80%.

* **Precision:** Precision quantifies the ability of a model to correctly identify positive instances among the ones it predicted as positive. It is the ratio of true positives to all predicted positives.
positive predictions to the total number of positive predictions (both true positives and false positives).

$$P = \frac{TP}{PredictedPositives} \qquad (A.2)$$

$$= \frac{TP}{TP + FP} \qquad (A.3)$$

Example: If a model predicts 100 instances as positive, and 80 of them are truly positive, the precision would be 80%.

Note that, with a model that almost always predicts the negative class, there will be minimal False Positives and True Positives ($FP, TP \to 0$), and thus, this flawed model would have nearly 0% precision.

* **Recall (Sensitivity or True Positive Rate):** Recall measures the ability of a model to correctly identify all positive instances in the dataset. It is the ratio of true positive predictions to the total number of actual positive instances in the dataset.

$$R = \frac{TP}{ActualPositives} \qquad (A.4)$$

$$= \frac{TP}{TP + FN} \qquad (A.5)$$

Example: If there are 100 positive instances in the dataset, and the model correctly identifies 80 of them, the recall would be 80%.

Note that, with a model that almost always predicts the positive class, there will be minimal False Negatives ($FN \to 0$), and thus, this flawed model would have nearly 100% recall.

* **$F_1$ Score:** The $F_1$ score is the harmonic mean of precision and recall. It balances the trade-off between precision and recall, penalizing disparities between these metrics, and provides a single metric to evaluate a model's performance.

$$F_1 = \left( \frac{1}{P} + \frac{1}{R} \right)^{-1} \qquad (A.6)$$

$$= 2 * \frac{P * R}{P + R} \qquad (A.7)$$

Example: If a model has a precision of 75% and recall of 80%, the $F_1$ score would be calculated as $2 * ((0.75 * 0.80)/(0.75 + 0.80)) = 77,4\%$.

## A.1 Evaluating Probabilistic Classifiers

A probabilistic binary classifier is a classifier that produces the probabilities of an item belonging to the positive class, rather than giving a definitive verdict. To take a decision, a specific threshold or cut-off is needed. For example, if a 50% threshold is chosen for the positive class, items with probabilities above this are deemed positive, while those below are considered as negative. Additionally, this threshold can be adjusted to be more stringent, requiring a higher confidence level (e.g., 60%) for positive classification, or more lenient, allowing lower confidence (e.g., 40%) for positive classification, and so on (Hanley and McNeil, 1982).
When evaluating a probabilistic classifier using a labeled dataset, different thresholds lead to varied confusion matrices. Using this insight, we can formulate a metric to evaluate such classifiers. To do this, we first need to understand two key concepts:

* The True Positive Rate (TPR) is the ratio of True Positive predictions to all actual Positive cases. That is, it's the recall (Equation A.5). It assesses the classifier's efficacy with respect to the positive portion of the dataset.

$$TPR = \frac{TP}{TP + FN} = R \qquad (A.8)$$

* The False Positive Rate (FPR) is the ratio of False Positive predictions to all actual Negative cases. It assesses the classifier's efficacy concerning the negative portion of the dataset, but in a reverse manner. That is, a higher FPR means worse performance regarding the negatives, and lower FPR means better performance (Equation A.10).

$$FPR = \frac{FP}{TN + FP} \qquad (A.9)$$

$$= 1 - \frac{TN}{TN + FP} \qquad (A.10)$$

The fundamental observation is that selecting a particular probability threshold gives rise to specific values in the confusion matrix, which in turn provides a pair of values (FPR, TPR). Plotting these points as the threshold varies creates the *Receiver Operating Characteristic* (ROC) curve.

To introduce the AUC measure, let's further explore the nuances of the ROC curve (Refer to Figure A.1):

1. A threshold of 0% implies that every item is classified as a positive, thus, there are no False Negatives and no True Negatives, which means that the FPR and TPR are both 1.

2. A threshold of 100% implies that every item is classified as a negative, thus, there are no True Positives and no False Positives, which means that the FPR and TPR are both 0.

3. Combining these two observations, if we plot the curve by decreasing the threshold from 100% to 0%, we will obtain a curve starting from point (0,0) and ending at point (1,1).

4. A perfect classification system would make no mistakes, no matter the probability threshold<fnref n="1" />. That is, it would have no False Positives and no False Negatives. This would make the TPR always 1, which corresponds to a horizontal line 1 unit above the horizontal axis.

5. A completely inaccurate classification system is always wrong, no matter the probability threshold. That is, it would never have True Positives nor True Negatives. This would make the TPR always 0, which corresponds to a horizontal line on top of the horizontal axis.

<footnote n="1">¹Except for thresholds of 0% and 100%, as explained above</footnote>
<figure><img src="image_74.png" /><figcaption>FIGURE A.1: Area Under Curve</figcaption></figure>

6. A classification system that always takes decisions at random, is equally likely to correctly identify positive instances as it is to incorrectly label negative instances as positive. This means, the *FPR* and *TPR* would always be equal, which corresponds to a diagonal line.

7. A classification system that performs better than a random classifier will have a performance tending towards a perfect classifier, which corresponds to a curve above the diagonal.

8. A classification system that performs worse than a random classifier will have a performance tending towards a completely inaccurate classifier, which corresponds to a curve below the diagonal.

The AUC is simply the area under the ROC curve, and can be used to evaluate probabilistic classifiers, because:

* A perfect classifier has an AUC of 1.0

* A completely inaccurate classifier has an AUC of 0.0

* A random classifier has an AUC of 0.5

* A better-than-random classifier has an AUC above 0.5

* A worse-than-random classifier has an AUC below 0.5
# Appendix B

## VSI Dataset samples and language statistics

This appendix presents tables and statistics to provide a deeper insight into the characteristics of the VSI dataset.

### B.1 Keywords for constructing the VSI Database
<table><caption>TABLE B.1: Keywords and key phrases used for the VSI dataset</caption><thead><tr><th>Organism of interest</th><th>Type of organism</th><th>Search Query</th></tr></thead><tbody><tr><td>Fusarium oxysporum f. sp. cubense tropical (race 4)</td><td>Fungus</td><td>fusarium oxysporum tropical</td></tr><tr><td>Xylella fastidiosa</td><td>Bacteril</td><td>xylella</td></tr><tr><td>Bursaphelenchus xylophilus</td><td>Nematode</td><td>Bursaphelenchus xylophilus</td></tr><tr><td>Bactrocera dorsalis</td><td>Insect</td><td>Bactrocera dorsalis</td></tr><tr><td>Candidatus Liberibacter spp.</td><td>Bacteria</td><td>huanglongbing</td></tr><tr><td>Popillia japonica</td><td>Insect</td><td>Popillia Japonica</td></tr><tr><td>Flavescence dorée</td><td>Bacteria</td><td>flavescence</td></tr><tr><td>Tomato brown rugose fruit virus</td><td>Virus</td><td>ToBRFV</td></tr><tr><td>Spodoptera frugiperda</td><td>Insect</td><td>spodoptera frugiperda</td></tr><tr><td>Bretziella fagacearum</td><td>Fungus</td><td>oak wilt Bretziella</td></tr><tr><td>Agrilus planipennis</td><td>Insect</td><td>Agrilus planipennis</td></tr><tr><td>Thaumatotibia leucotreta</td><td>Insect</td><td>Thaumatotibia leucotreta</td></tr><tr><td>Xylotrechus chinensis</td><td>Insect</td><td>Xylotrechus chinensis</td></tr><tr><td>Toumeyella parvicornis</td><td>Insect</td><td>Toumeyella parvicornis</td></tr><tr><td>Ceratocystis platani</td><td>Fungus</td><td>Ceratocystis platani<br>chancre du platane</td></tr><tr><td>General Plant Health</td><td></td><td>first report plant disease<br>new plant health</td></tr></tbody></table>
B.2 Sample entries detected as Latin

<table><caption>TABLE B.2: Sample entries detected as Latin text</caption><thead><tr><th>Text</th></tr></thead><tbody><tr><td>Bactrocera (Daculus) oleae (Gmelin, 1790)</td></tr><tr><td>resting on a leaf - Popillia japonica</td></tr><tr><td>La Popillia japonica devasta colture in Piemonte</td></tr><tr><td>Regione attiva piano anti Popillia japonica - www.lombardianotizie.online ...</td></tr><tr><td>Popillia japonica | Libero Quoditiano.it</td></tr><tr><td>Lombardy: anti Popillia japonica plan activated in Milan - Lombardy</td></tr><tr><td>Candidatus liberibacter ç¬¬1éµι</td></tr><tr><td>Milano, area di San Siro: Regione attiva piano anti Popillia japonica</td></tr><tr><td>A Milano il piano anti Popillia japonica | Milano.zone</td></tr><tr><td>Vithal Popillia japonica Polysect Ultra SL PFnPO Le Migliori Venditori</td></tr><tr><td>ultime notizie su ore popillia</td></tr><tr><td>Meridiem Seeds presenta Yuparanà nel Lazio</td></tr><tr><td>Lombardia: al via piano regionale contro la Popillia japonica a San Siro (2)</td></tr><tr><td>Milano, area di San Siro: Regione attiva piano anti Popillia japonica</td></tr><tr><td>Control of Anthracnose ...by Colletotrichum musae ...</td></tr><tr><td>...on Curcuma alismatifolia Gagnep using Antagonistic Bacillus spp.</td></tr><tr><td>Lombardia: al via piano regionale contro la Popillia japonica a San Siro</td></tr><tr><td>Popillia japonica , come difendersi</td></tr><tr><td>Japankäfer &lt;i&gt; Popillia japonica &lt;/i&gt;</td></tr><tr><td>RETI INSETTICIDE PER Popillia japonica - Asso Web TV</td></tr><tr><td>La Popillia japonica</td></tr><tr><td>Citrus tristeza virus ...of Candidatus Liberibacter Asiaticus by Diaphorina citri</td></tr><tr><td>Popillia japonica</td></tr><tr><td>Popillia devastante nel Santhiatese - Prima Vercelli</td></tr></tbody></table>
<table><caption>TABLE B.3: Number of entries per language in the VSI Dataset</caption><thead><tr><th>Language</th><th>Entries</th><th>Language</th><th>Entries</th><th>Language</th><th>Entries</th></tr></thead><tbody><tr><td>af</td><td>18</td><td>id</td><td>226</td><td>ro</td><td>77</td></tr><tr><td>am</td><td>2</td><td>ig</td><td>3</td><td>ru</td><td>180</td></tr><tr><td>ar</td><td>226</td><td>ilo</td><td>24</td><td>rw</td><td>13</td></tr><tr><td>az</td><td>13</td><td>is</td><td>34</td><td>sd</td><td>1</td></tr><tr><td>be</td><td>3</td><td>it</td><td>4788</td><td>sk</td><td>66</td></tr><tr><td>bg</td><td>107</td><td>iw</td><td>13</td><td>sl</td><td>51</td></tr><tr><td>bn</td><td>6</td><td>ja</td><td>74</td><td>sm</td><td>1</td></tr><tr><td>bs</td><td>33</td><td>jv</td><td>2</td><td>sn</td><td>8</td></tr><tr><td>ca</td><td>142</td><td>jw</td><td>1</td><td>so</td><td>4</td></tr><tr><td>ceb</td><td>7</td><td>ka</td><td>13</td><td>sq</td><td>18</td></tr><tr><td>co</td><td>121</td><td>kk</td><td>7</td><td>sr</td><td>12</td></tr><tr><td>cs</td><td>94</td><td>kn</td><td>4</td><td>su</td><td>1</td></tr><tr><td>cy</td><td>5</td><td>ko</td><td>51</td><td>sv</td><td>53</td></tr><tr><td>da</td><td>35</td><td>kri</td><td>5</td><td>sw</td><td>14</td></tr><tr><td>de</td><td>362</td><td>ku</td><td>3</td><td>ta</td><td>4</td></tr><tr><td>el</td><td>164</td><td>ky</td><td>1</td><td>te</td><td>9</td></tr><tr><td>en</td><td>12016</td><td>la</td><td>152</td><td>tg</td><td>12</td></tr><tr><td>eo</td><td>7</td><td>lb</td><td>28</td><td>th</td><td>31</td></tr><tr><td>es</td><td>3025</td><td>ln</td><td>3</td><td>tk</td><td>3</td></tr><tr><td>et</td><td>31</td><td>lo</td><td>4</td><td>tl</td><td>3</td></tr><tr><td>eu</td><td>3</td><td>lt</td><td>26</td><td>tr</td><td>83</td></tr><tr><td>fa</td><td>36</td><td>lv</td><td>11</td><td>ts</td><td>3</td></tr><tr><td>fi</td><td>25</td><td>mg</td><td>4</td><td>tt</td><td>1</td></tr><tr><td>fil</td><td>11</td><td>mi</td><td>5</td><td>uk</td><td>30</td></tr><tr><td>fr</td><td>2251</td><td>mk</td><td>1</td><td>ur</td><td>1</td></tr><tr><td>fy</td><td>12</td><td>mo</td><td>8</td><td>uz</td><td>11</td></tr><tr><td>ga</td><td>8</td><td>mr</td><td>19</td><td>vi</td><td>210</td></tr><tr><td>gd</td><td>3</td><td>ms</td><td>17</td><td>yi</td><td>1</td></tr><tr><td>gl</td><td>18</td><td>mt</td><td>6</td><td>yo</td><td>1</td></tr><tr><td>gn</td><td>69</td><td>ne</td><td>2</td><td>zh</td><td>41</td></tr><tr><td>gu</td><td>6</td><td>nl</td><td>465</td><td>zh-CN</td><td>784</td></tr><tr><td>ha</td><td>2</td><td>no</td><td>29</td><td>zh-TW</td><td>72</td></tr><tr><td>haw</td><td>3</td><td>ny</td><td>6</td><td>zu</td><td>4</td></tr><tr><td>hi</td><td>39</td><td>om</td><td>11</td><td>Failed</td><td>6132</td></tr><tr><td>hmn</td><td>2</td><td>pl</td><td>135</td><td></td><td></td></tr><tr><td>hr</td><td>77</td><td>ps</td><td>3</td><td></td><td></td></tr><tr><td>ht</td><td>4</td><td>pt</td><td>1292</td><td></td><td></td></tr><tr><td>hu</td><td>184</td><td>qu</td><td>4</td><td></td><td></td></tr><tr><td>hy</td><td>2</td><td></td><td></td><td></td><td></td></tr></tbody></table>

## B.3 VSI Language statistics

Table B.3 shows the distribution of languages (by ISO code) in the VSI dataset as automatically detected by the Google Translate API.
# Appendix C

## Regular expressions used in this project

In this Appendix, one will find illustrative examples of the regular expressions utilized for various tasks in this project. It is important to note that our matching algorithms are case-insensitive.

### C.1 Error Message Removal - VSI Dataset

<table><caption>TABLE C.1: Some Regular Expressions for Removing Errors Messages in the VSI dataset</caption><thead><tr><th>Regular Expressions</th></tr></thead><tbody><tr><td>~JavaScript is not available\.$</td></tr><tr><td>~Error$</td></tr><tr><td>~NA$</td></tr><tr><td>~Timeout error$</td></tr><tr><td>~None$</td></tr><tr><td>~Loading(\.)*$</td></tr><tr><td>~Not Found$</td></tr><tr><td>~Access Restricted$</td></tr><tr><td>~Page Not Found$</td></tr><tr><td>~\[\]$</td></tr><tr><td>~blacklisted</td></tr><tr><td>~'NoneType' object has no attribute 'get'?$</td></tr><tr><td>~PPlease Wait\.\.\s+ Cloudflare$</td></tr><tr><td>~HTTPSCConnectionPool.*</td></tr><tr><td>~Checking your browser.*</td></tr><tr><td>~Please update your browser$</td></tr><tr><td>~JavaScript n'est pas disponible\.$</td></tr><tr><td>~Vos données\.% Votre expérience\.*$</td></tr><tr><td>~Before you continue to YouTube$</td></tr><tr><td>~Just a moment(\.)*</td></tr><tr><td>~Your data\. Your experience\..$</td></tr><tr><td>~Access Denied$</td></tr><tr><td>...</td></tr></tbody></table>

### C.2 Noise Removal - VSI Dataset
<table><caption>TABLE C.2: Regular Expressions for Noise Removal</caption><thead><tr><td>Description</td><td>Regular Expression</td><td>Explanation</td></tr></thead><tbody><tr><td>Emojis</td><td>[\U00010000-\U0010ffff]</td><td>Unicode emojis</td></tr><tr><td>Hyphen-like Characters</td><td>[\u002D\u058A\u2010\u2011\u2012<br>\u2013\u2014\u2015\u2E3A<br>\u2E3B\uFE58\uFE63\uFF0D]</td><td>Common hyphen-like characters in<br>Unicode such as `--` ,<br>etc.</td></tr><tr><td>Quotation-mark-like Characters</td><td>[\u00BB\u00AB\u25B7\u2018\u2019<br>\u201C\u201D\u2039\u203A<br>\u300C\u300D\u300E\u300F<br>\u301D\u301E\u301F\uFE41<br>\uFE42\uFE43\uFE44\uFF02<br>\uFF07\uFF62\uFF63]</td><td>Common quotation-mark-like char-<br>acters in Unicode, such as » « ' "< ></td></tr><tr><td>URL</td><td>\bhttps?://\S+\b</td><td></td></tr><tr><td>Date</td><td>\b{4}[-/]{1,2}[-<br>/]{1,2}\b|\b{1,2}[-/]{1,2}[-<br>/]{4}\b|\b{1,2}[-/]{1,2}[-<br>/]{1,2}\b</td><td>Date pattern for dates in formats<br>YYYY-MM-DD, DD-MM-YYYY,<br>DD.MM.YYYY, DD/MM/YYY,<br>etc....</td></tr><tr><td>Hours</td><td>\b{1,2}:(?::{2})?\b</td><td>Hours pattern e.g. 10:30 AM,<br>15:45, or 2023-05-15 10:30:00.<br>Also, 15:45:30 and 2023-05-<br>15T12:30:45+00:00, etc</td></tr></tbody></table>