Regular Paper
SHUSUKE OHTAWA²,b) Ritsuko Oshima²,c)    Jun Oshima²,d) S HUNPEI Y AMAGUCHI 1,a) T AKASHI W ATANABE 1 , g ) T AKUYA F UJIHASHI 1 ,e) Shunsuke Saruwatari 1,f) Received: xx xx, xxxx, Accepted: xx xx, xxxx Abstract: Collaborative learning practices foster the ability to solve creative problems in collaboration with other learners. The collaboration enables learners to learn new ideas from other learners and enhances the social ability of the learners through interaction with other learners.  Although the learning science field now uses qualitative analysis to analyze the effects of the collaborative discourse, qualitative analysis requires much human and time costs to analyze the collaborative discourse with dozens of students. This study proposes Sensor-based Regulation Profiler to reduce the analysis costs. The proposed scheme consists of the business card-type sensors that acquire sensor data from each learner with a precise time synchronization as well as learning analysis methods that analyze the collaborative dis- course from the acquired sensor data. Experimental evaluations using the proposed scheme showed that the proposed business card-type sensors realized the time synchronization error of 7.7 μ s on average across the sensors. In addition, the proposed learning analysis could extract and visualize the collaborative activity of each learner in the collaborative discourse through the social graph extraction, learning phase extraction, speaker identification, and activity estimation by using the sensor data from the proposed business card-type sensors. Keywords: Collaborative learning, Human activity recognition, Sensor-based learning analysis, Time synchroniza- tion
Introduction
1.
Collaborative learning fosters the ability to solve challenging problems in collaboration with other learners.  Each learner can learn new skills and improve social skills through the collabo- ration with other learners.  The existing studies on collaborative learning find that distinctive interaction patterns increase the ef- fectiveness of the collaborative learning [17–21, 25].  However, the existing studies require a much time to find the patterns dur- ing the collaborative learning activity. In the existing studies, the researchers in learning science manually analyze each learner's activity from the recorded video and transcribed voice informa- tion. Such qualitative analysis is difficult to be carried out for the collaborative learning activity with a large number of the learners due to the amount of data and manual analysis, and thus the re- sults of the qualitative analysis are difficult to be immediately fed back to the learners. In this study, we propose Sensor-based Regulation Profiler to quantitatively analyze the activity of learners in collaborative learning and automatically extract and visualize the key points of the activity to support the qualitative analysis by the researchers in learning science. The proposed Sensor-based Regulation Pro- filer consists of business card-type sensors, namely, Sensor-based Regulation Profiler Badges and sensor-based learning analysis. The Sensor-based Regulation Profiler Badges acquire sensor data from the learners in collaborative learning.  In addition, each Sensor-based Regulation Profiler Badge has a RF-based time syn- chronization module to achieve high-precision synchronization of the sensor data between all the sensors. The proposed sensor- based learning analysis automatically analyzes and visualizes the key points of the collaborative learning activity from the acquired and synchronized sensor data. The proposed Sensor-based Reg- ulation Profiler allows researchers in learning science to analyze the collaborative learning activity in more detail through the au- tomatic extraction and visualization of the activity from the ac- quired sensor data.  We envisage the use of automatic extraction results to find groups whose collaborative learning activity is not working well and to help the researchers who navigate the learn- ing activity of those groups in real-time. From experimental evaluations using the proposed Sensor- based Regulation Profiler, we found 1) the average time synchro- nization error in the proposed Sensor-based Regulation Profiler Badges is approximately 7.7 μ s, 2) the proposed scheme automat- ically extracts social graphs that represent face to face interactions between learners, and 3) the proposed method also extracts learn- ing phases, speakers, and learner's activity during collaborative learning. For example, the proposed scheme accurately identifies speakers under different numbers of users, environmental noises, and reverberation conditions as well as short utterances. The remainder of this paper is as follows: Section 2 describes the requirements for introducing quantitative analysis to collabo- rative learning environments. In Sec. 3, we present the overview of our proposed scheme, namely, Sensor-based Regulation Pro- filer.  Sections 4 and 5 also present the details of the proposed Sensor-based Regulation Profiler Badge and sensor-based learn- ing analysis. Experimental evaluations are carried out in Secs. 6 and 7 to demonstrate the performance of the proposed Sensor- based Regulation Profiler Badge and sensor-based learning anal- ysis.  Section 8 discusses the related studies and Sec. 9 finally concludes our paper.
2.
Requirements
There are two requirements to deploy a quantitative analysis tool into collaborative learning activity. ( 1 ) Precise time synchronization across each learner's sensor data ( 2 ) Automatic extraction of the key points during collaborative learning The first issue is required for the extraction of interpersonal collaboration.  A main objective of this study is to analyze col- laborative learning activity from the business card-type sensors deployed on various targets such as learners and learning en- vironments.  There are several business card-type sensors such as Hitachi's business microscope [29, 30] and Massachusetts In- stitute of Technology (MIT)'s Sociometric Badge [31], Open Badges [11], and Rhythm [12].  For example, the business mi- croscope enabled the quantitative analysis of the communication in the organization to solve communication issues by analyzing the social networks between the members of the organization. However, the conventional business card-type sensors did not re- alize precise time synchronization across the business card-type sensors.  However, the studies have a drawback in precise anal- ysis of human collaboration in terms of synchronization across sensor data.  The studies attempt to synchronize sensor data by means of software correction. For example, the study [30] finds similar patterns in each sound pressure and synchronizes sound pressure data sampled at 8 kHz within 100 ms.  Pattern recog- nition further decreases the synchronization accuracy across the sensors at lower sampling rate for low power consumption. Such error causes inaccurate and meaningless analysis of collaborative learning activity.  To prevent the meaningless analysis owing to the synchronization error, the synchronization accuracy should be less than one-tenth of the sensor's maximum sampling rate. For example, if the maximum sampling rate of the sensor data is 100 Hz, the sensors should be synchronized with the time within 1 ms or less each other. The second issue is required to connect qualitative and quan- titative analysis in collaborative learning.  Ideally, quantitative analysis should automatically extract all of the same results as those obtained in qualitative analysis by the researchers in learn- ing science. However, it is difficult to replace all of the qualitative analysis with the quantitative analysis at present because there is a significant gap between the information that can be acquired by machines and humans. To fill the gap between the qualitative and quantitative analysis, our study focuses on automatic extraction of learning phases, social graph, speakers, and learner's activity. The extracted results can be used to detect speech segmentation during collaborative learning activity.  The detected speech sec- tions can help the part of the conversation analysis, i.e., the part of the qualitative analysis, by the researchers in learning science.
Proposed Scheme:  Sensor-based Regulation
3.
Profiler
In order to reduce the cost of qualitative analysis in collab- orative learning, we propose Sensor-based Regulation Profiler. The proposed scheme consists of Sensor-based Regulation Pro- filer Badge that acquires sensor data from learners and sensor- based learning analysis that extracts collaborative learning activ- ity from the acquired sensor data.  Figure 1 shows the overview of the collaborative learning analysis using the proposed Sensor- based Regulation Profiler. The Sensor-based Regulation Profiler supports qualitative analysis of collaborative learning by the re- searchers in learning science as following steps: ( 1 ) Distribute the proposed Sensor-based Regulation Profiler Badges to learners ( 2 ) Conduct collaborative learning activity between the learners ( 3 ) Collect the distributed Sensor-based Regulation Profiler Badges from the learners ( 4 ) Extract sensor data from the collected Sensor-based Regula- tion Profiler Badges ( 5 ) Automatically extract and visualize the social graph, learn- ing phases, and each learner's speech and activity from the sensor data ( 6 )  Carry out qualitative analysis by learning science researchers using the visualized results of social graph extraction, learn- ing phase extraction, speech detection, and activity estima- tion The proposed Sensor-based Regulation Profiler Badge is de- scribed in Sec. 4 and the proposed learning analysis methods are described in Sec. 5, respectively.
Sensor-based Regulation Profiler Badge
4.
Figures 2 (a) and (b) show the proposed Sensor-based Regula- tion Profiler Badge (Sensor node) and the block diagram of the Sensor-based Regulation Profiler Badge.  Fig. 3 shows Sensor- based Regulation Profiler Badge Synchronizer (Sync node). The Sensor-based Regulation Profiler Badge is a business card-type sensor designed to be worn on the chest of each learner. The sen- sor node consists of a power control unit, a CPU sensor unit, and a wireless unit. The power control unit has a lithium-ion battery to drive the sensor node. The lithium-ion battery supplies power to the power switch and the Micro Controller Unit (MCU). The sensor node can continuously run for 24 hours. The CPU sensor unit is equipped with STM32L476RGT6 from STMicroelectronics as the MCU, ADXL362 accelerom- eter from ANALOG DEVICES, OSI5LAS1C1A infrared light emitting diode (LED) from OptoSupply, PIC79603 infrared re- ceiver from KODENSHI CORP., and INMP510 analog micro- phone from TDK. The 3-axis accelerometer samples 12 bits at 100 Hz and the sound pressure sensor samples 12 bits at 100 Hz. The microSD card connector of DM3AT-SF-PEJM5 from Hirose Electric is used to record the sensor data. The acceleration data, infrared data, and sound pressure data can be recorded in a mi- croSD card. The wireless unit uses CC2650 from Texas Instruments which contains a wireless synchronization module. The wireless syn- chronization module transfers a synchronization signal transmit- ted every 10 ms from the sync node to other sensor nodes to real- ize time synchronization between the sensor nodes. The CC2650 uses UNISONet, which is also known as Choco [7, 28], to real- ize precise time synchronization between the sensor nodes.  In Choco, an arbitrary sensor node forwards a time-synchronous packet to the neighboring sensor nodes and then propagates the received time-synchronous packet to the destination node. When a sensor node receives a new time-synchronous packet from the neighboring sensor node, it immediately forwards the packet to the neighboring sensor nodes.  Each sensor node repeatedly re- ceives and forwards time-synchronous packets by flooding, re- sulting in fast propagation of time-synchronous packets through- out the sensor nodes.
5.
Sensor-based Learning Analysis
We also propose sensor-based learning analysis methods to automatically and precisely extract information of collaborative learning using sensor data obtained from Sensor-based Regula- tion Profiler Badges.  The proposed learning analysis supposes to support researchers in learning science to post analysis of col-
Algorithm 1 Social graph extraction Require:  L, U, to Ensure:  G l :  Insert zeros into all elements of G 2:  for all d ∈ U do S ← all received IDs in l d ∈ L between t 0 to + W 3: 4: for all s ∈ S do Increment G[s][d] 5: 6: end for 7:  end for 8: return G laborative learning. The proposed method realizes social graph and learning phase extractions from the infrared LED sensors, speaker identification from the sound pressure sensors, and activ- ity estimation from the three-axis accelerometers.
Social Graph Extraction
5.1
The social graph extraction visualizes social graphs that rep- resent the network of the learners in collaborative learning from the face to face relationship between the Sensor-based Regulation Profiler Badges.  The face to face relationship can be measured from the infrared data of each Sensor-based Regulation Profiler Badge. We note that the social graph extraction detects not prox- imity but face to face across the users. The infrared data contain the IDs of the other Sensor-based Regulation Profiler Badges de- tected every second.  Algorithm 1 shows the procedure of our social graph extraction and Table 1 shows the notation of the al- gorithm. Algorithm 1 outputs the matrix G which represents the social graph at to for L from the set of all sensor IDs U , the set of infrared data from all the sensors L = { l 1 , l 2 , . . , l 1 ( η }, and time instant to . The matrix G counts the number of infrared data across all the sensor nodes between to and to + W , where the row is the source sensor ID and the column is the destination sensor ID. For example, we consider that the infrared data of learner 1's Sensor-based Regulation Profiler Badge are as follows. Fig. 4: Example of social graph corresponding to the matrix G . 9000000, 1, 2 90000001, 2, 3, 2 90000002, 1, 2 Here, the infrared data consist of the time stamp, the number of other sensors detected by the infrared Light Emitting Diode (LED) sensor since the last time stamp, and ID of each detected sensor.  We also consider that the infrared data of learner 2's Sensor-based Regulation Profiler Badge are as follows.
90000002, 1,1 In addition, the infrared data of learner 3's Sensor-based Regula- tion Profiler Badge are as follows. 90000001, 1,1 In this case, the matrix G is given as follows:
The resultant matrix represents the directed graph shown in Fig. 4. Here, a larger value of the matrix indicates a higher fre- quency of face to face interaction between the learners and we use darker arrows in Fig. 4.
5.2 Learning Phase Extraction
The learning phase extraction automatically divides a collabo- rative learning period into learning phases based on the time vari- ation of the learners' network during collaborative learning. The appropriate number of learning phases depends on each case of collaborative learning [2, 4, 9, 13, 22, 24, 26, 27, 38, 39].  For ex- ample, the study [13] classifies learning activity into two phases: introduction and discourse and analyzes each phase. Our learning phase extraction supposes collaborative learning activities com- posed of three phases named as video viewing, discussion, and conclusion based on the study [2] which uses the same educa- tional material called the Adventures of Jasper Woodbury [5]. The video viewing phase obtains information from the video to solve the problem. The discussion phase is that the learners dis- cuss the problem and bring their ideas to solve the problem. The conclusion phase decides one answer from the ideas brought in the discussion phase. To divide the collaborative learning period into the learning phases, we regard the time variation of the learners' network as the transition of the learning phase.  At the transition of the learn- ing phase, distinctive points appear in the face to face relationship between the learners. For example, some learners may increase their attention to the different learners/objects or concentrate on a large number of the learners rather than the ones they were facing before.  We capture the distinctive points from the time varia- tion of the learners' network by using the infrared data of each learner's Sensor-based Regulation Profiler Badge. The time variation of the learners' network is quantified from the time variation of the matrix G . The matrix G of each time t 0 is extracted according to Algorithm 1. We consider that the matri- ces are extracted every 3 seconds and the window size W of each matrix is set to 60 seconds.  After created the matrix G for each window, it regards the residual sum of squares between the ma- trices of one window and the next window as the time variation of the network. We use AutoPlait [14] to quickly and automatically extract each learning phase from the network time variation.  AutoPlait detects the features from large time series data containing vari- ous patterns and regards the features as the groups of time series data. The proposed method discovers each learning phase as each group extracted by AutoPlait.
5.3 Speaker Identification
Figure 5 shows an overview of the proposed speaker identifi- cation algorithm. There are three steps for speaker identification: 1) pre-processing of sound pressure data, 2) speech section esti- mation, and 3) speaker identification. 1) Pre-Processing: The first step extracts the sound pressure detection for each user.  The algorithm calculates the minimum sound pressure value for each user and subtracts the minimum value from all the sound pressure data to make a zero-point cor- rection. The algorithm labels whether each user speaks with slid- ing windows for the sound pressure data of each user obtained by zero-point correction for each window.  Algorithm 2 exhibits the labeling procedure in Figure 5, and Table 2 lists the algo- rithm notation.  Algorithm 2 outputs the array A , which repre- sents "the 1–0 data for each user" from the set of all sensor IDs U and the set of the sound pressure data from all the sensors S = {S 1 , S 2 , . . . , S [U]} . We find the maximum of the sound pres- sure m for each user in each window W in line 6. If the maximum m in window W does not exceed the speech threshold η s across all users, it is assumed that the speech of the user is not detected in window W , and the window slides in line 16. If the maximum m in window W exceeds the speech threshold η s , the algorithm up- dates a threshold η m as m * 0.1 in line 8. The algorithm compares the sound pressure of a user with the threshold η m and assigns 1 if the sound pressure is higher than the threshold and 0 if the sound pressure is lower than the threshold in lines 9–13.  The labels w in window W overwrite the corresponding elements of array A d in line 14. We call the data obtained through pre-processing "the 1–0 data for each user." 2) Speech Section Estimation: The second step extracts the presence or absence of a user's speech from the 1–0 data for each user.  The algorithm fills the data using the 1–0 data for each
Algorithm 2 Labeling in pre-processing Require:  U, S Ensure:  A l :  for all d ∈ U do 2: Insert zeros into all elements of A s 3: ξ ← 0 4: while ξ < length of A do 5: W ⇐ S d ∈ S between ξ to ξ + D
user. The algorithm complements labels 1 in a section with con- secutive labels 0 within 90 ms between labels 1 considered in the middle of speech in the 1–0 data for each user.  The algorithm removes pulse noise using 1–0 data for each user with comple- ments.  The algorithm replaces a short interval with continuous labels 1 within 150 ms by labels 0, assuming that the section is where speech is falsely detected by ambient noise. The algorithm takes the logical summation of the 1–0 data for each user with pulse noise removal.  We call the binary data obtained through the speech section estimation "the speech section data." 3) Speaker Identification:  The third step determines who speaks in each speech section by combining the 1–0 data for each user and speech section data. The algorithm focuses on each sec- tion where a user is considered to speak based on the speech sec- tion data.  The algorithm extracts a user with the most labels 1 in each speech section and regards the user as a speaker in the speech section on the basis of the 1–0 data for each user.
5.4 Learner's Activity Estimation
Each learner's activity can be estimated from the accelerom- eter's data during collaborative learning. We first take L2- norm across three-axis accelerometer's data every sample in each Sensor-based Regulation Profiler Badge for the activity estima- tion motivated by the study on the human activity estimation us- ing the accelerometer [23]. Our proposed sensor mounts an ac- celeration sensor ADXL362.  ADXL362 quantizes and records acceleration within twice the gravitational acceleration. We sub- tract the offset in the data sheet of ADXL362 from all acceleration data to make a zero-point correction. We convert quantized accel- eration to relative values from 0 to 1 and visualize the values as learners' activity.
Experimental Evaluation: Time Synchro-
6.
nization Preciseness
We experimentally evaluated the time synchronization accu- racy between the sync node and the sensor node in the proposed Sensor-based Regulation Profiler Badge. We set up a sync node and a sensor node at a short distance on a desk and measured the time deviation between the nodes based on the synchroniza- tion signals sent from the sync node.  We used an oscilloscope to measure the clock rise time at each node to accurately get the time deviation between the nodes.  We assumed that the number of samples was 30,003 and the wireless synchronization module of each Sensor-based Regulation Profiler Badge transmits a syn- chronization signal every 10ms. Figure 6 shows the time synchronization accuracy between the nodes. The horizontal axis shows the deviation of the time syn- chronization and the vertical axis indicates the number of the samples corresponding to the deviation.  Fig. 6 shows that the time synchronization error is kept within ± 30 μ s. Here, the mean and maximum synchronization errors are −7.7 μ s and 30 μ s, re- spectively.  Since the sampling rate of both the sound pressure sensor and the acceleration sensor on the Sensor-based Regula- tion Profiler Badge is 100 Hz, the synchronization error is suffi- cient to meet the required synchronization accuracy of less than 1 ms. The proposed synchronization structure contributes to ac- curately analyze sensor data for collaborative learning.
7.
Experimental   Evaluation:
Sensor-based
Learning Analysis
We carried out collaborative learning experiments and evalu- ated the feasibility of the automatic extraction of social graph, learning phases, speakers, and activity with each learner's Sensor- based Regulation Profiler Badge during the activity.  Figure 7 shows a snapshot of our experiments in the collaborative learning activities. We carried out five cases of collaborative learning ac- tivities with three learners and monitored each activity who pos- sessed Sensor-based Regulation Profiler Badges.  We set a sync node at the center of the learners' desk for synchronization be- tween the sensor nodes.  Each learner mounted a Sensor-based Regulation Profiler Badge on his/her chest in case 1 and his/her head in cases 2 through 5 during the activity.  A whiteboard was set up to assist the learners in their discussions.  Two Sensor-based Regulation Profiler Badges were put on both edges of the white- board.  In addition, an iPad was placed on the desk to present learning tasks to the learners and one Sensor-based Regulation Profiler Badge was attached to the top of the iPad. 7.1 We evaluated the accuracy and validity of face to face detection with social graph extraction. We did an experiment with infrared sensors in our proposed business card-type sensors attached to three users to calculate the accuracy of face to face detection. We prepared the room with the dimensions of 10.6 m × 7.05 m × 2.65 m. The room furnished multiple LED recessed ceiling lights. Each user stood 1.50m away from the other users and two out of three users spoke face-to-face for 60 s.  A non-speaker faced the middle between two speakers during the conversation.  We tried all combination of speakers in the conversation and calculated the accuracy of face to face detection. We found that 1) the infrared sensors detect face to face with an accuracy of 75.3 %, 78.0 %, and 78.0 % in each combina- tion of speakers and 2) our proposed face to face detection suf- ficiently supports researchers in learning science to reduce quali- tative analysis cost of face to face in each experiment case. Fig- ures 8 (a) through (c) show social graphs in the learning phases in case 1 as an example. The learning elapsed time (s) is displayed at the bottom. In addition, Users 1 through 3 are the sensor nodes mounted on each learner, WB_R and WB_L represent the sensor nodes placed on the right and left edges of the whiteboard, iPad is the sensor node attached on the iPad, and the arrows show the face to face relationship across the sensor nodes. In Fig. 8 (a), we can see the learners did not face each other during the video viewing phase because the face to face relationship is scarce. Figure 8 (b) shows that User 1, User 2, and the right edge of the whiteboard faced each other and User 2 also faced the left edge of the white- board.  Since the learner closest to the right edge of the white- board was User 1, User 1 used the whiteboard for discussion and User 2 saw User 1's writing. In Fig. 8 (c), all the users faced the right edge of the whiteboard and Users 1 and 2 faced each other. The result said that User 1 used the whiteboard to conclude the activity and Users 2 and 3 saw User 1's writing. Qualitative anal- ysis requires researchers in learning science to repeat the recorded video and carefully annotate who and when learners met face-to- face. The proposed face to face detection reduces the process of watching the video and automatically extracts learners' face to face.
Results on Learning Phase Extraction
7.2
We evaluated the accuracy and validity of learning phase ex- traction in the collaborative learning activities. We simulated all combinations of window size and slide width for sliding windows in learning phase extraction by seconds and chose the best pa- rameters to calculate face to face difference across the users for learning phase extraction. We calculated the accuracy of learning phase extraction based on the qualitative analysis result as ground truth. Based on the design of learning phases in Sec. 5.2, we ex- tracted the best combinations of parameters for sliding windows in learning phase extraction from all combinations which output three phases with AutoPlait. Table 3 shows the best combinations of parameters and quali- tative/quantitative phase transitions in the learning phase extrac- tion.  Cases 1 through 5 accurately extract learning phases with the accuracy of 86.9 %, 100 %, 99.8 %, 91.1 %, and 90.9 % and predict the transition between learning phases within the error of about 1 min on average. We found that the results sufficiently sup- port researchers in learning science to reduce qualitative analysis cost of learning phase.  Figure 9 shows the results of quantita- tive and qualitative extraction of learning phases.  The top fig- ure in Fig. 9 shows the time variation of face to face difference across the learners during the learning activity.  The horizontal axis represents the elapsed time and the vertical axis represents the normalized time variation.  The three figures in the middle of Fig. 9 show the results of our learning phase extraction. The three figures show the duration of video viewing, discussion, and conclusion phase. The bottom figure in Fig. 9 shows the result of the qualitative extraction of the learning phases by the researchers in learning science. The result of the quantitative extraction indi- cates that 1) the learners do not often turn around owing to watch- ing the video in the video viewing phase, 2) the learners start to turn around to discuss the problem in the discussion phase, and 3) the learners often turn around to conclude the solution for the problem in the conclusion phase. The result shows the transitions between three phases appear in 1,202 s and 3,326 s. On the other hand, the result of the qualitative extraction shows that the transi- tions between three phases are from 1,173 s to 1,213 s and 3,335 s to 3,360 s. Although there are some deviations between the qual- itative and quantitative extraction results from 51 s to 403 s and from 2,289 s to 2,459 s, the automatic extraction accurately ex- tracts the transitions between three phases.
7.3 Results on Speaker Identification
We evaluated the accuracy and validity of our proposed speaker identification.  Our paper [37] shows the proposed system accu- rately identifies speakers under different numbers of users, envi- ronmental noises, and reverberation conditions as well as for long or short utterances. The experimental evaluations of collaborative learning show that the proposed speaker identification enables to streamline transcription of learners' utterance for collaborative learning analysis in each experimental case.  Figures 10 (a), (b), and (c) show the results of speaker identification in parts of the video viewing phase, the discussion phase, and the conclusion phase in case 1 as an example. The horizontal axis represents the elapsed time and the blue bars regard as the each learners' speech. Tables 4 (a) and (b) show the results of qualitative speech transcription in the sections of Figs. 10 (b) and (c). We extracted 60 seconds of the speaker identification result in each learning phase for simplicity.  Figure 10 (a) accurately detects no speech from 500s to 560s in the video viewing phase. The result shows that the learners did not speak to watch the video. Figures 10 (b) and (c) accurately extract each speech section from 1,300 s to 1,360 s in the discussion phase and from 3,700 s to 3,760 s in the conclusion phase. Qualitative analysis requires researchers in learning science to repeat the recorded video and observe who and when spoke as Tables 4 (a) and (b).  The proposed speaker identification reduces the process of watching the video and au- tomatically extracts speakers as Figs. 10 (a), (b), and (c).
Results on Activity Estimation
7.4
We evaluated the activity estimation of three learners in the col- laborative learning activity. Figures 11 (a), (b), and (c) show the estimated results of each learner's activity.  The horizontal axis shows the elapsed time and the vertical axis indicates the relative acceleration.  Tables 5 (a) and (b) show the result of qualitative records of learners' activity in the sections of Figs. 11 (b), and (c). We extracted 60 seconds of the same section as the speaker identification in each learning phase. Figure 11 (a) accurately de- tects small movements from 500s to 560s in the video viewing phase. The result shows that the learners did not move to watch the video.  Figures 11 (b) and (c) accurately extract each partic- ular movement from 1,300s to 1,360s in the discussion phase and from 3,700 s to 3,760 s in the conclusion phase.  Qualita- tive analysis requires researchers in learning science to repeat the recorded video and observe who and when moved as Tables 5 (a) and (b).  The proposed activity estimation reduces the process of watching the video and automatically extracts learners' particular behaviors as Figs. 11 (a), (b), and (c).
8.
Related Works
Our study relates to the studies on collaborative extraction us- ing business card-type sensors, sensor-based activity recognition, and collaborative learning analysis.
Collaborative Extraction Using Business Card-Type
8.1
Sensors
Several studies have tackled to extract the collaboration be- tween users using business card-type sensors on the users.  Hi- tachi proposes a business card-type sensor called Business Mi- croscope [29, 30] equipped with an infrared sensor which each user wears. The use of face to face information from the infrared sensors has shown that the appropriate frequency of meetings has an impact on work efficiency. MIT also proposes a business card-type sensor called Sociometric Badge [31] equipped with an accelerometer, a sound pressure sensor, a position sensor, Blue- tooth, and an infrared sensor which each user wears. Sociometric Badge collects face to face information between the users, con- versation tone changes, and proximity. The study [31] found that the face to face information between the users affects the work productivity and efficiency of the users. MIT extends Sociometric Badge to a small and low-energy sensor called Open Badges [11] with a sound pressure sensor and Bluetooth around the neck. Open Badges visualize face to face information between the users based on the sound pressure data and the Received Signal Strength Indicator (RSSI) from Bluetooth. MIT also develops hy- brid software-hardware platform called Rhythm [12] with Open Badges.  The platform measures face to face interaction in co- located contexts with Open Badges and in distributed contexts with their designed online applications. However, the studies have a drawback in precise analysis of human collaboration in terms of synchronization across sensor data.  The studies attempt to synchronize sensor data by means of software correction. For example, the study [30] finds similar patterns in each sound pressure and synchronizes sound pressure data sampled at 8 kHz within 100 ms. Pattern recognition fur- ther decreases the synchronization accuracy across our proposed sensors sampling sound pressure at 100 Hz for low power con- sumption. Such error causes inaccurate and meaningless analysis of collaborative learning activity. Considering the abovementioned drawbacks, we develop a novel business card-type sensor based on our initial study [1] to realize accurate synchronization across the sensors and learning analysis algorithm with the acquired sensor data.  The proposed sensor mounts the hardware structure to realize precise synchro- nization across the sensors. The proposed sensor receives and for- wards synchronization packets from its synchronizer across the other sensors.  The proposed sensor accurately acquires sound pressure, acceleration, and infrared data with synchronization across the sensors.  We have shown that 1) the proposed sen- sors achieve synchronization accuracy less than the error of 1 ms for acquired sensor data sampled at 100 Hz [32], 2) the learn- ing analysis algorithm extracts social graph, learning phases, and speakers [33–35], and 3) the algorithm improves the accuracy of Table 4: Qualitative transcription in each phase for case 1
speaker identification under various environments [36, 37].  This paper finally develops an IoT system with business card-type sen- sors for collaborative learning analysis. The proposed scheme ex- tracts social graph extraction, learning phase extraction, speaker identification, and activity estimation from the acquired sensor data. Experimental evaluations show the validity of each learning analysis algorithm in the collaborative learning activities.  Our proposed sensor and algorithms contribute to support qualitative analysis of collaborative learning by researchers in learning sci- ence.
8.2 Sensor-based Activity Recognition
Some studies have been carried out to recognize a user's be- havior by using multiple sensors attached to the user [3,6,8,10,
Number
12
15, 16].  In the literature [6], the sensor data obtained from at- taching accelerometers to the user's wrist, ankle, and chest are transmitted to the cloud. The cloud uses decision tree analysis to classify user's six activities: lying down, sitting, standing, walk- ing, running, and riding a bike.  In the literature [15], the user wears a wristwatch-type wearable device with built-in accelerom- eter, light sensor, thermometer, and sound sensor to classify the user's six activities: sitting, standing, walking, going up stairs, going down stairs, and running. They demonstrated that the clas- sification could be realized in real-time with 92.5 % accuracy by using decision tree analysis. Literature [10] uses Zephyr BioHar- ness Bluetooth to collect acceleration and biometric information on each user and then classify three activities of running, walking, and sitting by using the decision tree analysis.  In addition, they showed that the classification can cope with new users without re-learning by using the data of various users. Literature [8] used fuzzy basis functions for 3-axis accelerometer's values worn on the wrist of the user's dominant arm to classify seven activities: brushing teeth, tapping a person, tapping a desk, working on a computer, running, waving, and walking. Our proposed Sensor-based Regulation Profiler uses sensor data obtained from the proposed Sensor-based Regulation Profiler Badges to extract and visualize the key points during the collabo- rative learning activity. For example, the proposed Sensor-based Regulation Profiler automatically extracts the variations of the learning phases by measuring the network variation among the learners from the infrared sensor data mounted on each Sensor- based Regulation Profiler Badge. The automatic extraction of the learning phases can reduce the qualitative analysis cost of col- laborative learning as well as proper navigation in collaborative learning by the researchers in learning science.
8.3 Collaborative Learning Analysis
A general way to evaluate the effects of collaborative learning is to analyze the captured video and audio data corresponding to the activity of the collaborative learning [17, 19, 20]. On the other hand, the above-mentioned way has the following two issues. ( 1 ) The cost of detailed learning analysis ( 2 ) The inclusion of the researcher's subjectivity in the analysis The first issue stems from the transcription of the learners' con- versations from the captured video and audio data and the anal- ysis of the conversations considering collaboration using nonver- bal behaviors. The second issue stems from the misalignment of notation between the researchers when they analyze the learners' conversations. To solve the aforementioned issues, our study quantitatively analyzes the collaborative learning activity by using the proposed sensor-based learning analysis. The quantitative analysis of col- laborative learning realizes the detailed and quick analysis of the interaction between the learners.  In addition, our Sensor-based Regulation Profiler Badge can regard the learner's nonverbal be- haviors as the value of each sensor.  The proposed sensor-based learning analysis enables the analysis of collaborative learning activity without the researcher's subjectivity.
9.
Conclusion
We proposed Sensor-based Regulation Profiler to automati- cally extract and visualize the key points that researchers in learn- ing science pay attention to during collaborative learning activ- ity. Specifically, the proposed sensor-based learning analysis ob- tained social graph, learning phases, speakers, and activity from the acquired data of the proposed Sensor-based Regulation Pro- filer Badges.  Experimental evaluations show that our proposed Sensor-based Regulation Profiler Badge achieves the synchro- nization error across the sensors within ±30 μ s. In addition, our proposed sensor-based learning analysis extracts social graph, learning phases, speakers, and activity in collaborative learning. Each quantitative analysis reduces qualitative analysis cost of col- laborative learning by researchers in learning science.
Acknowledgment
This work was supported by JSPS KAKENHI Grant Num- ber 19H01714 and JST PRESTO Grant Number JPMJPR2032, Japan. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]