dataesr/scientific-paragraphs-categorization huggingface dataset:
The paper is structured as follows. Section 2 introduces formative assessment and emphasizes limits of prior TEFA initiatives. Section 3 describes the formative assessment system used as the data provider of our study, as well as the dataset. Section 4 details the analysis we conducted and gives the main results. Starting from these results, Sect. 5 proposes an orchestration model of formative assessment sequences implementing the two-votes-based process. Section 6 discusses the limitations of our study. Section 7 concludes and discusses future work.
The Dataset We conducted our analysis on data gathered from the use of Elaastic in higher education from 2015 to 2019. Until now, we collected 623 sequences conducted by 53 teachers where 1769 learners provided 8757 answers and performed 9256 peer ratings. A sequence is characterised by a learning context (i. e. face-to-face, distant or hybrid), the answers of the first and second votes, as well as the number of participants. For each answer, the following data are collected: the learner identifier, the content of the rationale, the score and the selected choice(s) when applicable. If the answer is a first vote, it is characterised by additional data such as the mean grade assigned by peers to the rationale associated with the answer, and the confidence degree of the learner who provided the answer.
Data Analysis The whole dataset has been filtered in order to reduce influential external factors and outliers. First, we only considered choice questions so as to be able to evaluate correctness of answers. In our analysis, in order to classify an answer as right or wrong, we considered answers as incorrect if the score is lower than the maximum score that can be obtained (i. e. 100). Also, since the asynchronous nature of distant and hybrid execution contexts in Elaastic doesn't require full orchestration from teachers  [30] , we kept face-to-face sequences only. Then we removed sequences where there were less than 10 participants because we wanted to focus on large scale settings. Finally, we considered the variables p1 and p2 which are the proportion of learners who answered correctly at the first and second vote respectively.
Limitations Our main limitations come from the dataset itself. The 104 sequences that we analysed addressed mainly STEM topics from higher education classes. A more refined study of sequences from various topics and educational levels could lead to broader findings. In the context of multiple choice answers, if a learner obtains a score of 33/100 during the first vote and 66/100 during the second vote, both her answers are considered as wrong, and the information stating that she improved is lost. Even though multiple choice questions are only a small portion of our sample ( 10%), a deeper study addressing this distinction would be a more adequate way to refine our results. Moreover, as stated earlier, Elaastic does not capture all learning interactions in a face-to-face context, thus making us unable to identify every decisive aspects of a formative assessment sequence such as its context (i.
Conclusion and Future Works This paper focused on formative assessment and emphasized the challenge of its application in face-to-face contexts. We introduced TEFA as the solution that emerged to perform face-to-face formative assessment and also introduced rich formative assessment processes generating a lot of meaningful interactions. Based on literature and on a dataset gathered from the usage of a two-votes-based process in an authentic learning context, we proposed to study these interactions to (i) highlight new understandings of formative assessment; (ii) provide system designers with evidences intended to help them to design a formative assessment system; (iii) identify meaningful indicators to assist teachers when orchestrating a face-to-face formative assessment sequence. Future works will implement our orchestration model within Elaastic while taking in account the explainability issues regarding our indicators.
Methodology While thousands of learning designs have been created by teachers in Go-Lab, to test drive our analytic approach we have chosen a set of learning designs that are publicly available (due to pragmatic concerns). Given the depth of (qualitative) analysis involved, we decided to analyze all published learning designs done by Estonian teachers (where the authors' research team is based). Analyses at the country level are especially relevant in Go-Lab and similar platforms, since this is the level at which most of the training and other actions are planned and implemented (i. e. , more direct practical implications). While this dataset of published Estonian inquiry designs is probably not representative of the typical pedagogical adoption of IBL in the Go-Lab platform, it represents, in theory, an interesting high-end reference subset of the inquiries designed by teachers in Go-Lab.
-49% of the phases present activities promoting active engagement (such as videos to be played or labs to manipulate), which the ICAP framework relates to shallow understanding; -68% of the phases contained activities envisioned to trigger constructive engagement (e. g. , asking learners to create a hypothesis, a concept map, make observations, extract conclusions, etc. ), which is associated with deep understanding and potential for learning transfer; -24% of the phases also include activities requiring interactive engagement (e. g. , collaborating or discussing ideas with other peers), thus aiming at a deepest understanding with potential for novel ideas. -Higher levels of engagement are rarer than lower levels in our dataset. Yet, looking at the learning design level, we can see that all analyzed designs reached the constructive level at least in one of their phases.
Interactive levels of engagement were only reached in a part of our dataset (66% of the designs). Regarding Lakkala's classification of social nature of learning  [13] , the results presented in Table  2  show that: -Most phases include either individual activities (e. g. , reading, playing a video, measuring, or interacting with a lab) or the individual creation of products such as hypotheses or conclusions. -Only 9% and 3% of the phases include collective activities or products (e. g. , a discussion or joint concept map, respectively). -At the phase level, practitioners are more prone to include activities than to request the creation of products. -At the learning design level, it is interesting to see that all designs include individual work, but close to 40% also involve some kind of collaboration in at least one of their phases.
Discussion While initially learning designs should aim at higher levels of engagement  [3] , exclusively using highly complex learning tasks would yield excessive cognitive load for the learners, having a negative impact on learning, performance, and motivation  [24]. Thus, the results of our case studyanalysis seem to align with the common approach of moving progressively from relatively simple to more complex learning tasks  [26] , requesting in each IBL phase different levels of engagement so that activities are complex enough to provide learners with new information and, on the other, easy enough so that they can integrate such information  [1]. Also, we can observe that the distribution of engagement modes varies across IBL phases in our dataset, fitting generally the kind of work expected in each type of IBL phase.
Even if the sample of learning designs under analysis may differ from the learning designs created by the average Estonian practitioner for use in their own classrooms, these findings lead us to think that it may be necessary to reinforce (e. g. , in teacher training events): 1) the added value of prompting interactive engagement; 2) the benefits of collaborative work; and 3) the importance of covering and guiding the discussion phases of an inquiry. To support teachers in this endeavour, the authoring platform could also integrate feedback and recommendations to take those aspects into account (e. g. , recommending interactive activities, collaborative apps, or explicitly including a discussion phase in teachers' designs). This illustrative case study does not come without limitations. First, the size of the dataset (3.
Related Work The research field around AQG has recently shifted from rule-based (e. g. [7] ) and template-based approaches (e. g. [10] ) towards neural network-based models  [5]. The main reason is the superior language generation capabilities of neural models compared to rule-based models  [5]. State-of-the-art systems apply large-scale language model pre-training before fine-tuning on SQuAD to improve the general English token prediction capabilities before predicting the task  [4]. In education, the LearningQ  [2]  and RACE  [8]  datasets are two large-scale datasets comprising questions with explicit educational intent. Furthermore, there have been studies exploring the educational use of neural question generation approaches in empirical studies  [6, 15, 16]. Besides the English language, there also exist AQGs in other languages.
For Japanese, using sequence-to-sequence learning and classical statistical learning techniques as AQG have been explored on a small-scale dataset with promising results  [12]. Another study uses the overgenerate and rank approach in Chinese, combining a rule-based system with a statistical ranking in a factual AQG, outperforming the rule-based system  [9]. Recently XNLG  [3]  has been proposed, a multilingual neural language model pre-trained in fifteen languages. For Chinese, it has been demonstrated that if XNLG is fine-tuned for AQG on SQuAD, it can generate plausible questions  [3]. However, to achieve those results, XNLG was pre-trained using computational resources often not available to researchers on a large-scale multilingual corpus  [3]. If the corpus does not contain the language target for generation, there is currently no way to import it into XNLG later.
Empirical Evaluation The evaluation study comprises two human annotators with a background in educational sciences, annotating 80 generated questions in random order. The questions are generated for 48 educational texts, written by teachers based on Wikipedia articles, as first used by Rüdian et al. [14]. We randomly select 26 texts with a total of 80 questions from the initial 48 articles. The dataset has question-worthy sentences in the texts already marked by educational experts. During data annotation, annotators apply the hierarchical scheme by Horbach et al. [6]. We achieve a average Krippendorff's α = 0. 35 over the nine categories. An overview of the results in the levels of the annotations scheme is given in Fig. 1. We report the results relative to the remaining questions and to all questions, since some questions will not be annotated fully, due to the hierarchical nature of the evaluation scheme.
Methods Data were collected from i-Learn, a prototypical technology enhanced learning portal which offers learning tracks with learning activities (e. g. exercises, communication tools, engagement activities, etc. ) from several educational applications. On i-Learn, teachers can support students, and assign them to groups and learning tracks. Our dataset consists of 803 students (228 from 5 primary schools, and 575 from 7 secondary schools), and 45 teachers from 12 schools who used the portal from November 2020 to March 2021. In total, 13,184 student events were logged. All students could rate activities on how fun an activity was, from 1 (not very fun) to 5 (very fun). Secondary school students could also rate on difficulty, from 1 (very easy) to 5 (very hard). From these events and ratings, 19 aggregate student features were extracted for clustering, resulting in one value per feature.
Hence, we were motivated to design and conduct a survey to analyse the usage of AR-EAs from the teacher perspective, as the related literature is rather thin. While the AR technology first emerged in the 1960s, the research work on deploying AR in education has only been published since 2000 (cf. the Scopus database). Given that students are the bigger target group (based on the teacher to student ratio alone), only a small number of studies explore exclusively teachers' views on deploying AR-EAs (e. g. [3] [4] [5] [6] ). All four studies had two common shortcomings: the teacher participants involved had no or little experience in using AR applications and the scale of the individual study was limited to a certain region of a single country. Meanwhile, several systematic literature reviews (SLRs) on AR-EAs have been conducted (e.
Methodology In order to qualify the current and expected teachers' uses of the LMS, we applied a quantitatively driven mixed method. We started with a quantitative analysis to deduce statistically different profiles of LMS use, in order to find groups of teachers or profiles of interest, based on the LMS log data. We performed a Principal Component Analysis (PCA) and a clustering analysis. PCA analysis allows to highlight diversity of the dataset in a reduced set of variables (components) while the clustering one aims to regroup the different instances of the dataset regarding their similarity. Afterwards, we conducted semi-structured interviews (i. e. : qualitative interview) with 3 female engineers on the same day (each lasted 40 to 50 min). In a series of open-ended questions prepared in advance to guide our interview, we collected information to improve the quantitative study which was analyzed by 2 researchers.
Methodology To conduct the research, we divided our work in different stages. Next, we explain each part of the methodology in detail: -Data extraction. The first step in our analysis was to get all the data necessary to begin the research. To download each paper in PDF, we used the Springer Link database  [11]. On the other hand, we used two different databases to get each paper's metadata: Scopus  [1]  and Web of Science  [10]. Although we did not find the metadata corresponding to the papers of the EC-TEL 2015 edition, we included these papers' metadata that we needed for our analysis manually. The final data collection contains a total of 447 documents and 1 905 keywords (4. 26 average keywords). -PDF parsing. In this stage, we parsed every PDF file into a plain text (TXT) file. To make that possible, we used PdfToText library, which parsed 100% of the papers with high fidelity.
Method We framed the study using design-based research (DBR)  [1] , a research method for studying learning in authentic contexts through interventions that often involves various stakeholders. The project was a collaboration between the developers of the screencasting tool, six schools, researchers and an experienced reference teacher. Together, the team developed the classroom intervention, which we tested in six secondary schools, in the school years 8 to 13, over the course of two academic years. Data in this paper is from the second design iteration. We performed two iterations of the intervention. In this paper, we focus on Scrimba as a code editor compared to Scrimba for recording student-produced screencasts. Six teachers and 134 students participated in the project. The dataset consisted of two different types of audio-visual recordings: video recordings of classroom activity (28 h) and screencasts produced by students and teachers (71 screencasts, in total 7 h).
Educawood exploits existing semantic datasets of land cover maps and forestry inventories that were released as Linked Open Data for the Iberian peninsula in our previous work  [1]. Teachers can check such information to propose contextualized activities for environmental education, e. g. finding a holm oak in a nearby dehesa and annotate it. Students can go on a field trip and perform the proposed activities, using Educawood with their mobile phones to annotate trees (locations, species, dendrologic measures, photos. ) as required. These annotations are published as Linked Open Data and can thus be reused for performing new learning activities such as virtual field trips.
A forestry inventory provides tree annotations of a territory using a sampling strategy, e. g. the Spanish inventory uses a grid of 1 km 2  cells. In our previous work we have proposed the tool Forest Explorer  [4]  for browsing those datasets. Tree annotation is an important activity in the forestry domain and in environmental education. We have thus developed the Simple Tree Annotation ontology (STA -namespace sta). It supports typical tree annotations -namely, location, height, width, and species identification-plus image annotation and creator metadata. We borrow terms from WGS84 Geo 1  and Dublin Core 2 vocabularies when appropriate. Note that STA supports multiple and probably inconsistent annotations from multiple users -the ontology includes primary properties, e. g. hasPrimaryPosition that can be used for conflict resolution.
Architecture of Educawood Educawood is a socio-semantic system for the annotation of trees and other ecosystem structures. The logical architecture is graphically depicted in Fig. 1. Users access the system through the GUI; this component exposes an interactive map for exploring forestry data coming from land cover maps, forest inventories, and social tree annotations. The GUI also includes a form for creating annotations such as the one in Listing 1. User actions performed through the GUI are forwarded to the Application logic component that provides the system's core functionalities: user management, tree annotation, forestry data retrieval, and image upload (which relies on an external Image cloud service). Tree annotations created with the system are stored in the Educawood tree annotations dataset.
Iberian land cover maps and forestry inventories are also employed to retrieve forestry data from an area of interest. Since these sources are available as Linked Open Data, we use CRAFTS 3  (Configurable RESTful APIs For Triple Stores). More specifically, the Educawood API exposes a regular RESTful API that greatly simplifies the access to the data sources -note that the alternative involves the usage of Semantic Web technologies such as SPARQL, OWL, and RDF. We have developed a preliminary working prototype of Educawood. The source datasets are already deployed in distinct SPARQL endpoints, while the Educawood API is deployed at https://crafts. gsic. uva. es/apis/educawood. The Application logic is based on Express 4  -a popular Node. js web application framework. We have employed Angular 5  to develop the GUI.
However, he likes the tasks available in CasualLearn and he has spent some time doing them, even if they are not mandatory. All this information is collected by the monitoring hub, and stored in a database. The acquisition of the data is performed through different monitoring adapters, that convert the requests to the specific API of each platform and translate the incoming data into a common format within SCARLETT. These traces are later processed in the student model and context manager. This module manages the instances of the student model (4), with variables devoted to track student progress associated to the learning goals specified in the LD (e. g. , knowledge of Romanesque), that evolve as action data is gathered and learning analytics are performed (e. g. , analysis of the grades in a quiz on Romanesque).
Technological Background Learning design authoring tools support teachers in creating learning activity plans, including description of tasks, supporting resources and tools and expected learning outcomes  [10]. Although there are several learning design tools available, only a few of them provide co-design features  [11] [12] [13]  and they have been essentially oriented towards supporting co-design between teachers  [14]. Our tool has been built upon the Ldshake  [11]  and the Ld-feedback  [15]  tools. Although the connection between the design tool and the feedback app was already explored in the past  [15] , in the current tool we are extending this connection to all the steps of the lesson plan and improving the integration between them. The current platform is realized as a website and -as is common for this type of application -consists of a frontend written in HTML, JavaScript and Jquery, as well as a backend with a MySQL database and PHP code.
Description of the Web-Based Application and its Use Cases The web-based weekly timesheets consist of a monolithic web application that was developed in Ruby under the Ruby on Rails framework (https://rubyonrails. org/) according to a Model-View-Controller design pattern (see Fig. 1 ). This web-based application runs inside a Docker container (https://docs. docker. com/compose/rails/), using a PostgreSQL database (postgresql. org). It uses Git for version control, and the source code is stored in a Bitbucket repository. The purpose of using web-based weekly timesheets is twofold. First, it makes the students aware of their weekly workload inside and outside the class because they are encouraged to estimate the hours spent on specific tasks -including class-attendance and self-study activities.
Acknowledgments. This project has received funding from the  European Union 's  Horizon 2020 research and innovation programme  under grant agreement No. 669074. Project " TU TEE -Tallinn University as a promoter of intelligent lifestyle " (No. 2014-2020. 4. 01. 16-0033 ) under activity A5 in the  Tallinn University Centre of   Excellence in Educational Innovation. Acknowledgments. We would like to thank our colleagues (E. B, M. S), the members of the school (D. S. , A. L. ), teachers (M. H. , S. K. , B. J-D. , Y. G. ) and students who helped set afoot the experiments. Acknowledgement. This work was financed by  Carnot TSN/IMT Atlantique  ( 2019-2020 ). Acknowledgements. I would like to thank the 377 first-year OU students who took part in the 'younger students' survey, and the OU strategy and student offices who contributed their insights and experiences to the instrument development and data analysis.
Acknowledgement. We would like to thank DUTE 2020/2021 students for granting permissions to collect data for this study. The  Universidad de Valladolid  co-authors would like to acknowledge funding of the  European Regional Development Fund  and the  National Research Agency of the Spanish Ministry of Science, Innovation and Universities , under project grants  TIN2017-85179-C3-2-R  and  PID2020-112584RB-C32 , and the  European Regional Development Fund  and the  Regional Government of Castile and Leon , under project grant  VA257P18. Acknowledgements. This research was funded by the  NFR IKTPLUSS  project  ALerT , # 270969  and Excited, The  Norwegian Center for Excellent IT Education. We thank the students who participated in the game co-creation sessions. Acknowledgement. This project is funded by the  CNED  which provides us with data for this work.
Acknowledgements. This research was supported by  NSF  Award # 1760922. We thank all the participating teachers. Acknowledgement. This work was supported in part by Grant # 1822861  from the  National Science Foundation (NSF). Any opinions presented in this article are those of the authors and do not represent the views of the NSF. We thank all participating teachers and the anonymous reviewers. Acknowledgements. This research has been partially funded by the  European Union  in the context of  CEITER  ( Horizon 2020 Research and Innovation Programme , grant agreements no. 669074 ). The authors would like to thank the rest of the coding team ( Aleksandr Trofimov ,  Jaanika Lukk ,  Katariina Vainonen , and  Mariell Miilvee ) for their contribution to this study.