Question: What is the two-step approach used by the authors to apply the identifier?
Evidence: We downloaded questions and answers via the Smartvote API. The downloaded data cover 175 communal, cantonal and national elections between 2011 and 2020.
All candidates in an election who participate in Smartvote are asked the same set of questions, but depending on the locale they see translated versions of the questions. They can answer each question with either `yes', `rather yes', `rather no', or `no'. They can supplement each answer with a comment of at most 500 characters.
The questions asked on Smartvote have been edited by a team of political scientists. They are intended to cover a broad range of political issues relevant at the time of the election. A detailed documentation of the design of Smartvote and the editing process of the questions is provided by BIBREF12.
We merged the two labels on each pole into a single label: `yes' and `rather yes' were combined into `favor'; `rather no', or `no' into `against`. This improves the consistency of the data and the comparability to previous stance detection datasets.
We did not further preprocess the text of the comments.
As the API does not provide the language of comments, we employed a language identifier to automatically annotate this information. We used the langdetect library BIBREF13. For each responder we classified all the comments jointly, assuming that responders did not switch code during the answering of the questionnaire.
We applied the identifier in a two-step approach. In the first run we allowed the identifier to output all 55 languages that it supports out of the box, plus Romansh, the fourth official language in Switzerland. We found that no Romansh comments were detected and that all unexpected outputs were misclassifications of German, French or Italian comments. We further concluded that little or no Swiss German comments are in the dataset: If they were, some of them would have manifested themselves in the form of misclassifications (e.g. as Dutch).
In the second run, drawing from these conclusions, we restricted the identifier's output to English, French, German and Italian.
We pre-filtered the questions and answers to improve the quality of the dataset. To keep the domain of the data surveyable, we set a focus on national-level questions. Therefore, all questions and corresponding answers pertaining to national elections were included.
In the context of communal and cantonal elections, candidates have answered both local questions and a subset of the national questions. Of those elections, we only considered answers to the questions that also had been asked in a national election. Furthermore, they were only used to augment the training set while the validation and test sets were restricted to answers from national elections.
We discarded the less than 20 comments classified as English. Furthermore, instances that met any of the following conditions were filtered from the dataset:
Question is not a closed question or does not address a clearly defined political issue.
No comment was submitted by the candidate or the comment is shorter than 50 characters.
Comment starts with “but” or a similar indicator that the comment is not a self-contained statement.
Comment contains a URL.
In total, a fifth of the original comments were filtered out.
The questions have been organized by the Smartvote editors into categories (such as “Economy”). We further consolidated the pre-defined categories into 12 broad topics (Table TABREF7).
The dataset is shared under a CC BY-NC 4.0 license. Copyright remains with www.smartvote.ch.
Given the sensitive nature of the data, we increase the anonymity of the data by hashing the respondents' IDs. No personal attributes of the respondents, such as their party affiliation, are included in the dataset. We provide a data statement BIBREF15 in Appendix SECREF8.
In the first run we allowed the identifier to output all 55 languages that it supports out of the box, plus Romansh, the fourth official language in Switzerland3 . We found that no Romansh comments were detected and that all unexpected outputs were misclassifications of German, French or Italian comments. We further concluded that little or no Swiss German comments are in the dataset; otherwise, some of them would have manifested themselves via misclassifications (e.g. as Dutch). In the second run, drawing from these conclusions, we restricted the identifier’s set of choices to English, French, German and Italian.
Answer: First allowed the identifier to output all languages and get the conclusions, second restricted the identifier’s set of choices drawing from these conclusions.