Development of a method generating criteria and standards of rubric
for problem-solving on VLEs
- From learner’s products for the assignment
Mariko Taga, Toshihiro Kita, Shirou Kitamura, Ryuichi Matsuba
Kumamoto University-Graduate School of Social and Cultural Science, Japan099g9901@st.gsis.kumamoto-u.ac.jp
kita@ield.kumamoto-u.ac.jp
kit@cc.kumamoto-u.ac.jp
matsuba@cc.kumamoto-u.ac.jp

Abstract:
We developed a rubric to enhance active learning on an online classroom of a graduate school. The analysis of thelearner’s products for assignment offers inspection of criteria and standards for the self-assessment in the rubric. In thepresent paper we show that a procedure to extract the criteria and standards for self-assessment based on outcomes of
their continuous learning activities and the effect of the self-assessment with use of the rubric. It is found that the newelements of the rubric checked the performance assessment for problem-solving learning improve the quality oflearner's outcomes, especially documents, for assignments required in the course.
Keywords: problem-solving, virtual learning environment, VLE, self-assessment, rubric, self-regulated learning

Introduction
Learners on Virtual Learning Environments (VLEs) are often isolated because they can see no presence of
co-learners, Teaching Assistants (TAs), teachers in asynchronous learning activities. Although Learning Management
Systems (LMSs) provide learners with bulletin board system (BBS) for communication tool, learners cannot have an
opportunity to examine, evaluate, or/and improve his/her work through observing others learning. Learning on VLE
is difficult because of asynchronous interactions with computer-mediated communication tools such as BBS cannot
feedback learners in timely manner.
Consequently the difficulty with online asynchronous learning is as follows.
Teachers and TAs:z cannot confirm the learner’s condition or state, which leads to fail to personalized guidance in timely manner.
z are difficult to confirm whether learners follow appropriate procedure or not.
Learners:z cannot obtain right answer or corrective action immediately.
z have little opportunity to enhance their learning through observing other learners working.
z tend to drop out in early stages of semester because of insufficient academic support.
z assume teachers provide little feedback with their work in the period of learning.
z are passive to receive assignments as “a mere task” with little concerns to promote cognitive strategies.
This study addressed the effect on the self-assessment using rubric for promoting self-regulated learning against thedifficulty with learning of problem-solving on VLE as above.
The advantages of using self-assessment are as follows:
Teachers and TAs:z enable learners to assess their performance with specific standards on courses period of time, then enhancelearner’s achievement.
Learners:z have clear understanding of goals in terms of self-assessment.
z obtain appropriate learning guidance that leads to reduce drop out rate of the course.
z can easily know differences between expected and actual achievements with on-going self-assessment.
z understand what is essential to improve their performance.

The target course is compulsory subject offered at Kumamoto University-Graduate School of Social and Cultural
Science. The course offers problem-solving, collaborative learning and VLE. Certification of credit is required tolearner offering a product for assignment of assuming corporate setting, learners are required to develop proposal tomeet request for proposal (RFP) of curriculum development for new employees of a virtual corporation. Items
described must be included in the product for assignment is specified by teacher. The analyzed data are nineteenproducts for assignment.

Purpose of the ResearchThis study addressed effects on the self-assessment using rubric for promoting self-regulated learning against thedifficulty with learning of problem-solving on VLE as above. Participants were graduate students who have various
academic backgrounds and have many social experiences. These heterogeneous conditions promoted meaningfullearning regarding PBL using mini-case.
On the other hand those conditions sometimes results in collision of meaning to interpret the requirements ofexercises assigned on the course. We aim at to prevent learners from misunderstanding the stem, which lead to
re-work or drop out.
Research question is: How can we make learner understand themselves their mistakes and lack of skills in theacademic requirements clearly?

Literature Review
Definition of rubric
Arter & Chappius define the rubric which is one of the mechanisms that determines the quality of learner’s tasks1). Rubric is
structured as matrix with performance criteria on vertical axis and levels should be achieved by each performance criteria on row
axis. Tanaka(2003) mentioned use of rubric to grade according to standards for qualitative changing point of learner’s cognitive
activities for learning subjects, and describes samples of product or anchors7).
Huba & Freed insisted the meaning of evaluation using rubric, quoted Locker as bellows3).
“Learning is promoted when learner themselves notice what they learn, recognize the standards described clearly, and recognize
the methods understanding what is they learned. Assessment demand educational providers the performance standards withverbalism expressed clearly”.
Evaluation for individual activities as “the self intension about specific items”, refers to levels that achieved will structures asfollows:
Level 1: He/she cannot lead to a conclusion.
Level 2: He/she leads to a conclusion without data based on the fact.
Level 3: He/she leads to a conclusion with data based on the fact, but they cannot mention to prove the conclusion.
Level 4: He/she leads to a conclusion with data based on the fact, and they mention to prove the conclusion.
Arter & Chappius mentioned the facilities by using rubric as bellow1):
1) Rubrics help teacher clarify themselves what ambiguous learning targets look like. They help learners do the same.
2) Well described rubric that includes clear performance criteria and standards is worked as the assessment for learners.

Procedure of rubric development
Linn & Gronlund offer four categories as general standards to describe the learning targets using rubric, that is to say
performance criteria6).
1) Definite contentions, arguments and composition
2) Embodiment, Specified
3) Exquisite main subject
4) No lexical error
In addition, they mentioned that it should do following procedures to determine the performance criteria and standards for eachlevel should be achieved. The table 1 is procedure for rubric development that Linn & Gronlund insist, arranged by Taga.
Table１ procedure for rubric development by Linn & Gronlund
Step
Details
1)
Reviewing the learning targets assigned each lecture. By review, the learning targets and instructions will
be accommodated standards for grading.
2)
Introducing attributes should be included/not included in the product. By this work, the skills and the
behaviors mastered through the learning will be observed.

Table１ continued
Step
Details3)
Expressing idea for explain the each attributes. The method should be explained clearly for the excellent/average/poor performance for observed each attribute introduced at step 2.
4)-a
As holistic rubric, describing complete details about excellent and poor performance for each attribute.
Connecting up descriptions for whole attributes to describing both the highest and lowest class’s
performance levels.
4)-bAs individual rubric, describing complete details about excellent and poor performance for each attribute.
Describing the details for that suitable grade. Prompts or encouragements for learners should be includedin the described details.
5)-aAs holistic rubric, determining the edges the highest/lowest levels for the holistic attributes.
And describing the middle range level.
5)-bAs individual rubric, determining the edges the highest/lowest levels for the holistic attributes.
And describing the middle range level.
6)
Collecting samples of learner’s products to instance each level.
7)
Revising rubrics on needs.
Evaluating the result of rubric at classroom. Revising before next semester.
(Linn & Gronlund(2000) 6) arranged by Taga)

The elements including rubric for performance assessment
Jonassen(2004)5) mentioned that rubric should develop addressing both product and process. Jonassen(2004)4) quotes
Elliott(1995) 2):
Elliott insists that performance assessment includes these elements.
z Learner must construct a response or a product rather than simply select from a set of predefined alternatives or answers.
z Assessment consists of direct observation or assessment of student behavior on problem-solving tasks.
To those, Jonassen add a third:z Assessment of quality of the product or observation using a rubric, that is, a description of desirable performance.

Self-regulation at cognitive strategies
Zimmermann mentioned there are three stages of self-regulation: Planning, Execution, Introspection8). On social cognitive
theory, Execution stage is composed by three sub-stages: Self-monitoring, Self-judgment, Self-reaction. Self-recording helps
Self-monitoring9).

MethodTarget course and analyzed data
The target course is compulsory subject offering at Kumamoto University-Graduate School of Social and Cultural
Science. Course objectives are to acquire knowledge and skill regarding problem-solving through collaborative
learning activities on VLE. Participants are nineteen graduate students who had experienced practices.
Teachers gave learners a case which describes a corporate situation. Learners are required to submit proposal of
employee’s performance improvement for technical boot camp of the corporation. Components of proposal werepre-defined by teachers and the course material on the VLE gave learners instruction to include specific topics
related the components in their products assignment. The data analyzed were documents submitted for course
assignments.
Viewpoints for developing criteria of rubric
Linn & Gronlund 6) mentioned the criteria described in the rubric is “learning targets and attribute include in it”, asprior chapter.
The learning target of problem-solving belongs to cognitive domain. According to Bloom cognitive performance
criteria are grouped into six categories: Knowledge, Comprehension, Application, Analysis, Integration and
Evaluation. Problem-solving requires to be conducted all six kind of performance including analysis, integration andevaluation. Therefore, performance criteria should be required not only Knowledge or Comprehension level but also
next step of learning, so-called, Application, Analysis, Integration and Evaluation.
The target course is designed as a learning sequence that learner should collect data on the case that presented,
analyze problems, consider solutions, estimate various essential resources, and offer the feasible planning sheet as
document finally. We aim rubric has elements based on the structure of product required in the course assignments.

The performance criteria and level achieved are evaluated by decoding the attributes included in the product, andrelation, strength and weakness between them.
Concept of rubric development for continuous learning activities at problem-solving
We developed the concept of method of generating criteria and standards for rubric as self-assessment for
problem-solving based on the “sequence of learning activities” that learners should take. The method is as follows:(1) List up items that should be described in the product for assignment by instruction.
(2) Relate these items with each other.
(3) Identify activities what should to perform the task required. These activities that found are “true learning
activities”.
(4) Re-compose these activities into “true sequence of learning activities”.
(5) Categorize the sequence of learning activities as “the performance that should be assessed”.
(6) Integrate/organize collections of performances that should be assessed into “holistic skills will be achieved”
that is to say “learning target of the course/curriculum”.
(7) Offer the standards for each level achieved as each activity in the learning sequence.
(8) Offer the anchor for the standards as the product which is made at the activity in the learning sequence.

Figure 1 Concept of rubric development for continuous learning activities at problem-solving
Analytical processes of products for assignment
We used to data analysis method in order to develop achievement’s grade. The procedure for the analysis is:(1) Obtain nineteen products for assignment with digital data.
(2) Segment product into a unit of phrase in relation to chapter.
(3) Code every segmented data with appropriate labels.
(4) Examine if the description of the segmented data is related to other description of the product.
The rating rules are:a. There is no description about the subject that must be included by instructions.
b. There is a description without the fact based on case offered.
c. There is a description without coherence on the case offered to mention to prove the conclusion.
d. There is a description with coherence on the case offered to mention to prove the conclusion.
(5) Deform achievement level for each unit of learning in terms of the rating of step (4).
Implement technique of development rubric
Actual technique for making rubrics is as follows:(1) Describe or illustrate whole sequence of events in the course. Make the material that teacher would offer andproducts learners should submit clear.
(2) Make the items and indexes of product to be submitted should be clear. Describe or illustrate flowchart of thesequence of events in the lecture.
(3) There may be several products of learner’s work. Divide a sequence into several sub-sequences each product.
Make only one “re-arranged sequence” for one product. Each criteria of rubric for self-assessment corresponds
these re-arranged sequence.
(4) By step above, only one product is corresponded only one sequence of events. Several products may correspondto one sequence of activities, constraint on which meets the condition that the sequence of learning activities
needs all products for progress to next sequence.
(5) Set different criteria, if identify a fork of the sequence of events.
(6) Consider the loop of sequence of events whether the whole of loop can regard as one sequence. For example,
the sequence of learning activities is an analysis from various points of view to obtain appropriate resolve forthe problem that found by needs analysis(7) Arrange the standard achieved of the lowest level is the first activity of the sequence of learning activities, andthe standard achieved of the highest level is the last activity of that one.

Results and Discussion
Formative evaluation
We did the formative evaluation by questionnaires to learners and interviews to teacher. Based on analysis on the
responses of the questionnaires, we concluded that offering rubrics in the course as self-assessment on
problem-solving in VLE is effective in advance promote learning, and self-judgment of self-regulated learning.
These opinions from the questionnaires to learners categorized evaluation indexes by social cognitive theory as
follows:(1) Encouragement of Self-monitoringz I used it as rating of perfection at developing the product for assignment.
z Rubric is useful for checking if there was lack of consideration or description.
z It is useful not only check-list but also interactive evaluation.
z To continue such learning and guidance, it may progress with learner’s achievement.
z Rubric is useful to learners who have scant experience in this field.
z Rubric is useful to follow up learning appropriate such VLE, it includes difficulty to interact each learnersynchronously.
z Self-assessment is important both VLE and face-to-face classroom. It is meaningful to train method orconcept about assessment.
(2) Encouragement of Self-Judgmentz I used it as self-checking before tendering the products for assignment.
z Rubric offers criteria and standards for learning goals and levels achieved.
z Rubric is useful to learn critical viewpoint.
(3) Encouragement of Self-Reactionz First I couldn’t use it effective. When feedback from teacher has returned, I have understood the meanings ofthe rubric how to use.
z I have understood that I should pass judgment on samples offered on the course too.
These opinions from the interviews to teacher directed our attention to sequence of the learning activities:
z There were differences of grading between teacher’s assessment and learners self-assessment for the products
for assignment. The learners almost tend to grade their products higher than teacher.
z Teacher’s impression was the learners tried to carry on their learning by their own interpretation without truecomprehension or confirmation that the tasks required.
z Teacher guessed those results were the reasons why the rubric first we developed was based on only
instructions and items described must be included in the product for assignment and there was no concreteanchor in the rubric.

Figure2

配属先

研修担当者

「日経文庫に書いてある内容を聞かされて退屈」と訴える新入社員も２割程度いた。
入社後の研修で損害保険、生命保険や業界についてゼロから講義をせざ
研修担当者
るを得なくなり、講義はかなりの詰め込みになってしまう。
基礎的な部分については一定数の新入社員が飽きていることが明らか
研修担当としては、内定者がどの程度学習しているかといった状況が把握できないため、学習の督促などのフォローもできず、また、研修計画時にどこまでを前提知識として有しているかも分からないため研修計画に際しても非常に苦慮しており、何らかの方法で学習の進捗状況を把握したいと考え
研修の講師を務める社員から、業界や保険についてのごく基礎的なこと 配属先（用語や保険のあらましなど）は入社前に理解させておいて欲しいという要
望が、年々強くなっている。
商品の多様化、業界内の競争激化、保険金未払い問題、コンプライアンス
強化等に伴い新入社員導入研修で学ぶべきことが年々増加し、研修時間 研修部門の不足が深刻になってきたためである。

講義はかなりの詰め込みになっている。
基礎的な部分については一定数の新入社員が飽きていることが明らかで内定者がどの程度学習しているかといった状況が把握できない。
学習の督促などのフォローもできない。
研修計画時にどこまでを前提知識として有しているかも分からない。
研修計画に際しても非常に苦慮している。
何らかの方法で学習の進捗状況を把握したい。
業界や保険についてのごく基礎的なことは入社前に理解させておいて欲し
いという強い要望がある。
従来からの研修内容は、配属先の上司からしっぱり学ばせて欲しいという要望が高い。
従来から研修内容として取り上げてきたものは保険商品・金融商品の知
識、提案営業のための顧客分析、リスクマネジメント、プレゼンテーションな商品の多様化、業界内の競争激化、保険金未払い問題、コンプライアンス
強化等に伴い新入社員導入研修で学ぶべきことが年々増加している。
今年は研修期間は2ヶ月（4月1日から5月31日）になった。（従来より１ヶ月

Analysis Needs, ステークホルダー 現状分析
ステークホルダー 現状分析
指定図書の理解をしていない内定者は多い。
入社時点で日経文庫「損害保険の知識」「業界研究シリーズ生保・損保」日
then Finding the
全く読んでいない者も少なくない。
経文庫ビジュアル「保険の基本」の内容を理解していない内定者は多く、そ 新入社員
研修についていくのが大変という意見もあった。
れどころか全く読んでいない者も少なくなかった。
指定図書の内容を聞かされて退屈と訴える新入社員も２割程度いた。
problems 新入社員 「研修についていくのが大変」と日々の受講アンケートの感想欄に書いてい入社後の研修で損害保険、生命保険や業界についてゼロから講義をして現状の研修内容

外部環境・市場動向

新入社員の行動保険商品・金融商品の知識
提案営業のための顧客分析
リスクマネジメント、プレゼンテーション

講義は詰め込みになっている

基本的なことについてゼロから講義

コンプライアンス強化

保険金未払い問題

業界内の競争激化

商品の多様化

基礎部分は一定数の新入社員が飽きている

指定図書の内容を聞かされて退屈である

まったく課題図書を読まない者もいる

研修期間

外部環境・市場動向

現状の研修内容

新入社員の行動研修リソース

研修内容への要望

研修期間

問題あり
研修内容への要望

研修リソース

変更や制約あり

現状の研修内容

マッチしているか

外部環境・市場動向

新入社員の行動

Level 1
Level 2
Level 3
Level 4
Level 5Extracting the data, writings regarding Subdividing writings into simple
Classify simple sentences as categories Relate each classified category with Leading the issues from related
needs from assignment case.
sentences.
of needs.
other categories.
categories of needs.
ニーズを抽出できている状態。
データが細分化できている状態。 カテゴリ分類が出来ている状態。 カテゴリ同士の統合（関連付け）が出 課題が導き出されている状態。
「抽出する」とは、元のデータをそのま 「細分化する」とは、元データを「文」の 「分類する」とは、細分化したデータが 来ている状態。
カテゴリに統合したニーズから、課題
まの形で、表などでまとめることであ 単位で切り出すことである。
示していることが何に関するものなの 抽出したカテゴリが示している事項に や問題点、確認すべき事項を短い言る。
「文」にするには、接続詞や句点（、）の か識別することである。
着目し、他のカテゴリでの事項と関連 葉で表す。
元データは、「文章」として表現されて 箇所で文章を分断する。
識別した結果、似たもの同士を示す共 があるか見極める。
いる。
さらに、分析の結果、一度に複数の事 通の言葉｢カテゴリ」を見つける。 あるカテゴリと親子関係になるかなど、柄が「文」に含まれていないか見直す。 そして似たもの同士にまとめる。 判断する。

To implement rubrics developed by our insist is supposed hopeful to obtain the effect that both learners andteachers can confirm the absolute improvement for individual level achieved, or can sense the fulfillment as alearner.

Sample of self-assessment
As described prior section, we extracted elements of rubric and used them to conduct self-assessment on the course.
Figure 2 illustrates an example of self-assessment worksheet, its criteria and standards were derived from theextraction procedure.
We have considered series of the learning actions of analysis need as follows:(1) Extracting the data, writings regarding needs from assignment case.
(2) Subdividing writings into simple sentences.
(3) Classify simple sentences as categories of needs.
(4) Relate each classified category with other categories.
(5) Leading the issues from related categories of needs.

Sample of self-assessment（Analysis of needs）

The new element of rubric for performance assessment
As mentioned at literature review, Jonassen5) quotes Elliott(1995) 2):
Elliott insists that performance assessment includes these elements.
z Learner must construct a response or a product rather than simply select from a set of predefined alternatives or answers.
z Assessment consists of direct observation or assessment of student behavior on problem-solving tasks.
To those, Jonassen add a third:z Assessment of quality of the product or observation using a rubric, that is, a description of desirable performance.

We would like add a fourth, modestly:z Assessment of quality of the product or observation using a rubric for cognitive domain for example
problem-solving, the criteria and standards should be developed based on the sequence of activities what thelearners should perform their required tasks to lead to their self-regulation.

Conclusion
We developed a procedure to extract the criteria for self-assessment based on outcomes of their continuous learning
activities, and examined the effect of the self-assessment by using rubric.
By the results of the analysis, we introduced hypothesis that self-assessments based a sequence of the learning
activities using rubric are effective in advance promote learning, and developed the method to generate criteria andstandards of rubric from the products for the assignment.
The outcomes of this study are as follows:(1) By formative evaluation, the hypothesis as above, self-assessments using rubric for self-assessment based aseries of the learning activities is effective to advance promote learning.
(2) Based on this target course, we have developed a procedure of rubric development to introduce performance
criteria and standards as self-assessment for problem solving. For example, to resolve the learning processes of
the course into tasks based a series of the learning activities to place these tasks as the standards, then to present
each middle product by each tasks as samples or anchors.
Further research is required in order to address questions about the components of successful online PBL using our
rubric, and determine whether our rubric can promote development of learner’s self-regulation.

References
1) Arter and Chappius, “Creating & Recognizing Quality Rubrics” (pp.27-28)
Merrill/ETSCollege Textbook Series (2007). ISBN-13: 978-0-13-513420-72) Elliott, S.H. (1995). Creating meaningful performance assessments. ERIC Digest E531.
(ERIC document reproduction service No. ED 381 985)
3) Huba, M. E., Freed, J. E.:”Learner-Centered Assessment on College Campuses". Allyn and Bacon. Boston.
(2000).
4) Jonassen,D.H, “Learning to Solve Problems” (pp.147-150): Pfeiffer
Jone Willy & Sons，Inc (2004). ISBN 0-787906437-95) Jonassen,D.H, “Learning to Solve Problems” (pp.155-158): Pfeiffer
Jone Willy & Sons，Inc (2004). ISBN 0-787906437-9
6) Linn, R.L., Gronlund, N.E.: "Measurement and Assessment in Teaching Eighth Edition". Prentice-Hall, Upper
Saddle River, New Jersey. (2000).
7) Tanaka,K “Consideration of objective evaluation”, Kyorinkan, Tokyo (2003）
http://www.shinko-keirin.co.jp/csken/pdf/51_03.pdf 2008/04/22 retrieved8) Zimmermann.B.J. and Schunk.D.H “Self-Regulated Learning: From Teaching to Self-Regulative Practice 1stedition(1998)” translated by Tsukano (pp.1-13) Kitaooji-Shobou (2007)
ISBN978-4-7628-2579-8
9) Zimmermann.B.J. and Schunk.D.H “Self-Regulated Learning: From Teaching to Self-Regulative Practice 1stedition(1998)” translated by Tsukano (pp.140-141) Kitaooji-Shobou (2007)
ISBN978-4-7628-2579-8