# A Study of Measurement of Mentoring Activities Using Text Mining Technology

Kaori Namba, Toshiyuki Sanuki, Tetsuo Fukuzaki, Kazuhiko Terashima

► To cite this version:

Kaori Namba, Toshiyuki Sanuki, Tetsuo Fukuzaki, Kazuhiko Terashima. A Study of Measurement of Mentoring Activities Using Text Mining Technology. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.15-20, 10.1007/978-3-031-43393-1_2. hal-05533312

HAL Id: hal-05533312

https://inria.hal.science/hal-05533312v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# A Study of Measurement of Mentoring Activities
Using Text Mining Technology

Kaori Namba¹✉, Toshiyuki Sanuki¹✉, Tetsuo Fukuzaki¹✉,
and Kazuhiko Terashima²✉

¹ IBM Japan, Ltd., Tokyo 103-8510, Japan
{Knamba,fukuzak}@jp.ibm.com, toshiyuki.sanuki1@ibm.com

² Machida Technical High School, Tokyo 194-0035, Japan
terashima@machidath.ed.jp

**Abstract.** P-TECH is a pioneering education reform initiative to prepare young people with the academic, technical, and professional skills required for 21st-century jobs and ongoing education. Mentoring is one of the key activities of P-TECH for students to think about their pathways and build engineering attitudes. After each mentoring session, we take surveys to evaluate how the sessions affected the students and we achieved our objectives. In addition to conventional method, we executed systematic analysis of the responses to the open-ended questions using text mining technology. The subjects of this paper were 30 high school sophomores in 2019. We mentored them eight times in 2019 and 2020 and took surveys after each session. We gathered 1784 documents in total as the responses to open-ended questions and analyzed them with a text mining tool. The tool showed the top three frequent nouns were synonyms for “myself”, and the top three correlated nouns of “myself” were “future”, “thought”, and “way”. This indicates many students considered their pathways during the mentoring sessions and discussed them in their surveys. The tool also showed that the top correlated noun/verb phrases were “build... Plan”, “obtain... Advice”, and “have... Interest”. This indicates that students correctly received the messages intended by the mentoring. Our results showed that the mentoring activities helped students consider their future. Text mining was a useful technique to analyze answers to open-ended survey questions.

**Keywords:** P-TECH · Survey · Measurement · Mentoring · Text mining

## 1 Introduction

The social demand for career planning and development of high school students is growing as they prepare to succeed in new occupations [1]. Mentoring, the act of helping and giving advice to a younger or less experienced person, is well recognized as an effective method for working people or students to develop their career or skills. In the business field, the progress of a mentee is measured by tracking their individual development plan since mentee’s development objective and goal are clear, and each mentee has some level of maturity to describe own plan.
In the case of high school students, mentoring is a relatively new approach to develop their career or skills and the method to measure the efficacy of mentoring activities is not well established. It is usually measured by the analysis of student's responses to a predefined questionnaire at each mentoring activity. However, this is not sufficient for a schoolteacher to identify insights regarding the effects on student's thinking or motivation under personal conversation between mentor and mentee [2].

We propose a new method to analyze the efficacy of mentoring activities for stu-
dents by using text mining technology to investigate effects on student's thinking through
mentoring. As a part of the P-TECH pilot program in the Tokyo Metropolitan Govern-
ment [3], we conducted a two-year mentoring program for high school students, and we
applied this method with a prototype system based on the IBM Watson Discovery plat-
form, to analyze the text data of student's responses during this period. We successfully
identified effects in student thinking through the mentoring.

## 2 Conventional Method

Mentoring is well recognized as an effective approach for working people and there are activities to apply mentoring into career development for students [4]. The experiences of working adults in actual fields are useful student mentees in high school to help them plan and decide their future career.

Evaluation of mentoring is mainly performed through self-evaluation. It is suitable for a mentee to evaluate their growth by themselves. Objective-based evaluation is also required to measure effectiveness of mentoring activities, and surveys are often used [5]. A survey usually consists of two types of questions: quantitative questions and opened ended questions. Evaluation of quantitative questions is often performed with statistical analysis, requestion analysis, and so on. On the other hand, evaluation methods for open-ended questions are not well established yet. In most cases, the teacher or mentor needs to read through the text of the response of each question, requiring careful understand and insight of what each student is thinking. This analysis approach is subjective and takes efforts, especially with large volumes of data from multi-year mentoring activities.

The answer to an open-ended question is considered *text data*, i.e., natural language. It used to be challenging for a computer to analyze text data. With recent progress in artificial intelligence (AI) technology, a computer can now analyze text such as student's responses regarding their mentoring activities. This paper introduces and discusses the usefulness of text analysis technique, known as *text mining*, to analyze qualitative changes in student's thoughts about mentoring activities.

## 3 Proposed Method

### 3.1 Text Mining

Text mining is a process to extract useful information from text data with Natural Lan-
guage Processing (NLP) technology. Using NLP, sentences are divided into parts-of-
speech (nouns, verbs, adjectives, etc.) and useful information is extracted by analyzing
the frequency of word occurrence and correlation. Text data is a typical case of quali-
tative data, and the goal of text mining is to collect high value-added information from
qualitative data.
<figure><img src="image_4.png" /><figcaption>Fig. 1. Analysis steps of text mining</figcaption></figure>

Text mining steps are shown in Fig. 1. Significant effort is generally required to manually gather, pre-process, and transform data. This is one reason why text mining is not commonly used by non-experts.

## 3.2 IBM Watson Discovery

IBM Watson Discovery (IWD) [6] is a platform that enables intelligent search and text mining for large volumes of text data. IWD can gather text data from independent sources within a company and/or from the internet and extract hidden meaning from this data. IWD provides many useful features. We used the following:

* **Frequency:** indicates how many documents contain a given word.

* **Trend:** indicates a trend of the frequency of a word.

* **Collocation:** indicates simultaneous occurrence of certain words in a sentence.

* **Correlation:** indicates how relevant a word is to documents that match a search condition. The correlation value measures the level of uniqueness of the high frequency as compared to other documents.

IWD almost automates Step 1 to 3 from Fig. 1. It also provides an easy and intuitive analysis tool for Step 4. IWD brings the power of text mining to non-experts.

## 4 Tokyo P-TECH

P-TECH stands for Pathways in Technology Early College High Schools. P-TECH 9–14 School Model is a pioneering education reform initiative to prepare young people with the academic, technical, and professional skills required for 21st century jobs and ongoing education [7]. To provide a holistic approach to education and workforce development, IBM, the New York City Department of Education, and The City University of New York designed and launched the first P-TECH school in Brooklyn, New York, in September 2011—and the first class graduated in June 2015.

In Japan, there are three P-TECHs in 2022. Tokyo P-TECH [8] is the first P-TECH in Japan for Machida Technical High School and Nihon Kogakuin College Hachioji Campus, started in April 2021. Three IT companies partner with the two schools to help students with three years of Information Technology courses at the high school, and with two years of Network & Security courses at the college.

Starting in April 2019, we conducted a trial run prior to the official Tokyo P-TECH for Machida Technical High School. Several volunteers from the partner company gave IT lectures, and provided job shadowing, on-site teaching, and mentoring.

We conducted eight mentoring sessions for 30 students by ten mentors from IBM Japan. We visited the school for the first four sessions and the remaining sessions were conducted remotely due to the COVID-19 pandemic.
We asked all 30 students to fill out questionnaires after each mentoring session. Each survey consists of quantitative and open-ended questions. The quantitative questions ask about student satisfaction and/or enjoyment on a 5-point scale. The open-ended questions ask the students to describe knowledge acquired in the session, future challenges, and so on. This paper analyzes student answers to these open-ended questions.

# 5 Analysis and Result

## 5.1 Data

Each survey has multiple open-ended questions. We treated each answer as one *document*. We collected 2,021 documents from eight mentor surveys and entered them into Watson Discovery. Then, we excluded 237 trivial documents that essentially meant “Not Applicable” as this could introduce noise into the results. The following results are based on an analysis of the remaining 1784 documents.

## 5.2 Analysis Result

The top ten nouns are shown in Fig. 2 (a). The word “myself” is the most frequent and prominent noun. Runner-ups are nouns related to a pathway (“pathway”, “job”) and studying (“study”, “qualification”). It means many students wrote in the surveys about themselves and their future. The top ten verbs are shown in Fig. 2 (b). “think” is the most frequent and prominent verb, and “consider” is the third place. We consider this reflects that both the teachers and the mentors repeatedly insisted on the importance of thinking at both mentoring sessions and usual classes.

<figure><img src="image_5.png" /><figcaption>Fig. 2. The frequency of (a) nouns and (b) verbs</figcaption></figure>

Then, we focused on the most frequent noun, “myself.” Fig. 3 shows the top ten collocated (a) nouns and (b) verbs with “myself”, sorted by correlation. The nouns “future”, “way”, “pathway”, and “job” have the highest correlation, and all indicate future pathways. This aligns with the high correlations of “consider” and “think in Fig. 3” (b) show that many students considered their pathways during the mentoring sessions and discussed them in their surveys.
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Collocated (a) nouns and (b) verbs with “myself”, sorted by correlation</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Collocation between nouns and verbs, sorted by correlation</figcaption></figure>

Figure 4 shows collocation between general nouns and verbs, sorted by correlation. High correlation combinations indicate goals of mentoring, like “build… Plan”, “obtain… Advice”, “have… Interest”, etc. This indicates that students correctly received the messages intended by the mentoring.

We applied trend analysis from various perspectives to understand student growth over time though the mentor sessions. We did not identify any clear trends. For example, Fig. 5 shows the trend of mentoring keywords, i.e., Career (“pathway”, “job”, “college”, etc.), Study (includes “study”, “class”, “programming”, etc.), and Future (includes “future”, “objective”, “dream”, etc.). We expected that these words would be mentioned more in later sessions, but this is not what the data show. The word “pathway” is the most frequent in the sixth mentoring session, since it was the time for the students to decide their pathway whether entering further education or taking a job. It may be that the current set of questions is not suited for the investigation of student growth. We are considering future enhancements to our surveys.

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** Trend of mentoring keywords</figcaption></figure>
# 6 Conclusion

We used text mining to analyze the responses of open-ended survey questions after Tokyo P-TECH mentoring activities. As predicted, our results show that the mentoring activities helped students consider their future. Text mining was a useful technique to analyze answers to open-ended survey questions.

In future work, we would like to identify the relationship between numeric indices and the answers to open-ended survey questions. Additionally, we plan to modify the survey questions to better clarify student growth through mentoring.

# References

1. U.S. Bureau of Labor Statistics: "Career Outlook", https://www.bls.gov/careeroutlook/2015/article/career-planning-for-high-schoolers.htm. Last accessed 28 Feb 2022

2. Heppen, J.B., Zeiser, K., Holtzman, D.J., O'Cummings, M., Christenson, S., Pohl, A.: Efficacy of the check & connect mentoring program for at-risk general education high school students. J. Res. Educ. Effect. 11(1), 56–82 (2017)

3. Tokyo Metropolitan Government: Press Release https://www.metro.tokyo.lg.jp/tosei/hodoha ppyo/press/2019/04/23/18.html

4. Akili, W.: Mentoring engineering students: challenges and potential rewards. In: 121st ASEE Annual Conference & Exposition, pp. 1–13. Indianapolis, Indiana (2014)

5. Akerele, O., Vermeulen, A., Marnewick, A.: Determining the benefits of the engineering mentoring programmes for graduates. In: Proceedings of the International Conference on Industrial Engineering and Operations Management (2019)

6. Watson Discovery: https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-about. Last accessed 28 Feb 2022

7. Learn about P-TECH schools: https://www.ptech.org/about/. Last accessed 18 Feb 2022

8. Tokyo P-TECH: https://www.mext.go.jp/content/20210210-mxt_syogai01-100003289_4.pdf. Last accessed 18 Feb 2022