# Exploring the Dominance of the English Language on the Websites of EU Countries
## 3 Methodology
### Crawling Websites and Recording Relevant Information  
In the next step of our process, a number of websites equal to the sampling size were analyzed using a proprietary crawler which was developed using PHP and manipulated data in the MariaDB database developed during Step 1. The crawler first checked every available version of each website (http/https, with/without www). Priority was given to the https versions and the versions with www. The process followed the appropriate instructions available in the robots.txt file in order to determine whether crawling for that specific website was allowed or not. If it was, then the frontpage was scraped and analyzed. If not, the website was removed from the sampling pool and replaced. Additionally, if the page allowed crawling but returned an error page or a redirect it was also removed from the sampling pool and replaced. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Country** & **Websites** & **Sampling Size** \\ \hline Austria & 172,815 & 4063 \\ Belgium & 173,471 & 4063 \\ Bulgaria & 13,445 & 3178 \\ Croatia & 21,603 & 3489 \\ Cyprus & 1126 & 887 \\ Czechia & 221,473 & 4084 \\ Denmark & 180,513 & 4067 \\ Estonia & 24,452 & 3556 \\ Finland & 98,232 & 3992 \\ France & 475,661 & 4125 \\ Germany & 1,627,339 & 4150 \\ Greece & 79,116 & 3953 \\ Hungary & 99,081 & 3993 \\ Ireland & 47,360 & 3825 \\ Italy & 458,727 & 4123 \\ Latvia & 18,779 & 3406 \\ Lithuania & 40,485 & 3773 \\ Luxemburg & 9304 & 2876 \\ Malta & 605 & 529 \\ Netherlands & 628,465 & 4133 \\ Poland & 334,824 & 4110 \\ Portugal & 45,026 & 3809 \\ Romania & 88,885 & 3975 \\ Slovakia & 75,469 & 3943 \\ Slovenia & 27,787 & 3619 \\ Spain & 234,791 & 4088 \\ Sweden & 221,795 & 4084 \\ United Kingdom & 480,802 & 4125 \\ \hline
**Total** & 5,901,431 & 102,018 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of available websites and appropriate sampling size per country. In addition to that, the number of internal or subdomain links in each frontpage was counted and if that number was below 2 or above 250 the webpage was considered an extreme case and was removed from the sampling pool and replaced. The reasoning behind this is that, more often than not, low link pages are just placeholders and extremely high link pages are suspicious as they are often used to confuse search engines or are part of a back-link generation scheme. Additionally, they would increase the time required for the process to run without any clear benefit to the results. First the lang attribute of the html tag was recorded if available. Then the crawler attempted to detect the actual written language of the page using the PHP language detection [43] library. The language detection function processed the HTML DOM using PHP's DOMDocument parsing library. With its help, it retained the text from each element of the frontpage while removing elements that don't traditionally contain written content such as <head>, <script>, <style>, <svg>, <img>, and <code>. The full text of the page was then run through the language detection algorithm with a setting of 9000 max Ngrams in order to achieve more confidence in the result. If the full text of a page did not exceed 150 characters, any language detection was considered unreliable due to the short string and as such the language detection function returned an empty result. Afterwards, in order to detect whether another language is available on the website, the crawler scraped every internal or subdomain link present in the frontpage and used the same language detection function to determine the language of each website. The HTML tag's lang attribute of each subpage was also recorded. If a subpage's detected language or lang attribute was different to the frontpage's the crawler inserted this page into the database. These records would subsequently be used during the next step of the process to investigate the number of languages that a website supports. A decision was made to stop the crawling of the website after one level beyond the frontpage.