Outline of the Study on Secondary Analysis of Results of the National Assessment ofAcademic Ability

1. Purpose and summary of the study(1) Purpose of the study
The purpose of the study is to develop methods of utilizing the survey results by carrying out
a secondary analysis of the survey results over multiple years, and to demonstrate the findings
obtained from the secondary analysis. The first National Assessment of Academic Ability wasconducted in FY2007 while the 10th was implemented in FY2017. During this period, the survey
results have been published every year, and various additional analyses based on the survey
results have been carried out. The Curriculum Research Center of the National Institute for
Educational Policy Research has also conducted school-visit surveys, detailed analysis of theanswers to free-response questions, and analysis of the survey results over multiple years so as
to contribute to further use of the surveys and survey results by the schools and boards ofeducation.
(2) Summary of the study
In this study, research was conducted focusing on the following two points:(i) A case study on effective measures to improve academic ability by the schools and boardsof education.
(ii) Basic research for the construction of a database of the survey results and for various
secondary analyses.
For (i), the aim was to conduct a survey through visiting the schools and boards ofeducation and to identify effective measures to improve overall academic ability in termsof gaining a broad view of the changes taking place over time in the survey results, andutilizing the survey results. By capturing the genuine efforts being made to further improveeducation in the face of the various difficulties confronting the schools and the boards andeducation, and analyzing these findings to come up with suggestions, the study gives
specific examples of ways to utilize the survey results. In addition, the case examples ofschools that were visited were compiled in a separate booklet as the third collection of
already published case collections.
For (ii), various secondary analyses were attempted such as a detailed analysis of theanswers to free-response questions, correlation analysis between the results of the survey
relating to the subjects and the results of the questionnaire, correlation analysis between
the subject of Japanese language, and of mathematics and arithmetic. This study

demonstrated the findings obtained from a secondary analysis of the survey results, andexplored which secondary analysis methods were possible.
[Period of the study: FY2015 – FY2016, Research Project Leader: Atsushi Umezawa(Director of the Curriculum Research Center)]
2. Outline of the study outcomes(1) Study on utilization of the survey results
In this study, a detailed review was conducted of the relevant items and values for theresults of the survey relating to subjects and the results of the questionnaire. As a result,
the three perspectives given below emerged. Based on these perspectives and other factors,
the aim was to identify the actual situation of efforts that contribute to improving andenhancing the curriculum guidelines by first identifying and then visiting the “schools thatare producing results” in order to conduct a survey.
First perspective: Schools that have achieved results through the school’s own efforts.
Schools with a high rate of financial aid for school attendance, schools with a low cramschool attendance rate, schools with a small percentage of the C and D quartile levels.
Second perspective: Schools that have managed to resolve problems that werecontinuously seen in each of the subjects.
Third perspective: Response rate for items in the questionnaire relating to students’attitudes towards learning, etc.
Focus on the percentage of answers which were “strongly positive” for items concerning
motivation for learning and self-esteem.
As a result, what became apparent was that the following factors were effective in
improving the academic ability of the students: an environment where all the teachers andstaff look squarely at the situation the school is in, and with a clear understanding of theproblems, share this recognition amongst themselves and work tenaciously on the sameproblems, an environment where the teachers, with the cooperation of the students’ families,
tirelessly assess the growth of the students and encourage their further growth; anenvironment where the teachers carefully note where particular students are getting stuck
and work on their understanding by having them take small steps; and an environment
where students teach and learn from each other, and teachers teach and learn from eachother.
(2) Study relating to secondary analysis of the answers to free writing questions
In the survey relating to the subjects of the National Assessment of Academic Ability, thenumber of possible answer types was limited to a maximum of 10 for each question. Therefore,even if the answers were grouped into the same answer type, on closer examination, there wasan assortment of answers that could be further broken down. In addition, the primary purpose
when analyzing the answers was completion in a single fiscal year. Therefore, in this study, thequestions related to quotes from elementary school Japanese language and questions related toratios in elementary school mathematics were selected from the free writing questions, and thediversity of the answers was inspected and the trends of answers over multiple years wereexplored in detail.
As a result, even if answers were grouped into the same answer type, a more detailed
classification showed the challenges faced by the students in greater detail. This suggests theimportance of examining the design of classification that could lead to better teaching based on
an analysis of errors. Even with the same correct answers, there were differences in the qualityof the answers. This suggests that the same thing can be said even about the errors, giving rise to
the need to develop questions and establish answer types to better understand the quality ofstudents’ understanding and where they are getting stuck.
(3) Expectations of future analysis in light of the efforts in this study
Since further analysis seems to be needed in the future based on the results of the analysistested in this study and the issues, consideration will be given to further utilizing the results of
the questionnaire and to conducting a quantitative analysis of the results of the free writing
question.
The correlation coefficient between the respondents’ correct answer rate (number of correctanswers) and answers (positive / negative) to individual questionnaire items was calculated, anda regression analysis carried out, but deeper understanding was not reached on how the results of
the quantitative analysis could be used by the boards of education and schools. In this study, wefocused on the strongly positive answers of specific items of the student questionnaire when
visiting the schools to conduct the survey, and were able to refer to these answers when gauging
the efforts of the school, but it is thought that by increasing the items covered by the study,identifying questionnaire items that will more clearly show the efforts of the school, and looking
at the survey results over multiple years, it will be possible to identify the correlations between
long-term trends and the school’s efforts, and the effect of these on the percentage of questions
answered correctly in the subjects. In addition, if in the future, one idea is also to analyze thesurvey results of the National Assessment of Academic Ability, which is conducted as a complete
enumeration survey, by linking them to various other surveys, an analysis should be conductedin the future linking them to school surveys such as the School Basic Survey and the National
Survey on Physical Fitness and Exercise Habits, or surveys which clarify the family or regionalsituation such as the National Census, and consideration should be given to organizing the dataand refining the tasks accordingly.
In terms of conducting a quantitative analysis of the free writing questions, aside fromanalyzing the details given in the free writing with regard to issues that can be seen to be going
beyond the subjects or fiscal years, the aim was also to identify certain trends through quantitative
analysis. However, one major obstacle was that the design of the questions and the compositionof the answer types differed, which meant that the analysis did not lead to a clear conclusion. Inthe future, one idea is to refine the issues that need to be analyzed once again and to study thedesign of questions that go beyond the framework of the subjects.