Studiebarometeret: Rapport 5–2015

Personal feedback and advising in
Norwegian higher education:
Explaining student dissatisfaction
Stephan Hamberg, Marie-Louise Damen & Pål Bakken

Summary
The Norwegian Agency for Quality Assurance in Education has completed an annual student survey
about the perceived quality of education in Norwegian bachelor and master programs. Based on thenational student survey and interviews with students, faculty, and program leaders at seven different
study programs in medicine and engineering in Norway, we show that while students are dissatisfied
with the feedback and advising they receive, this dissatisfaction do not affect the students’ overall
perception of the quality of their study programs. We argue that the main reason for this is thatstudents do not expect formative feedback and individual advising to play a major role in their
university education.

iTable of content
1

Introduction ....................................................................................................................... 1

2

Background ....................................................................................................................... 2

3

Research Design ................................................................................................................ 3
3.1

4

5

Case selection .............................................................................................................. 4

Analysis .............................................................................................................................. 7
4.1

Feedback ...................................................................................................................... 7

4.2

Individual advising ...................................................................................................... 9

4.3

Overall satisfaction .................................................................................................... 10

4.4

The missing link between feedback, counselling and overall satisfaction ................ 12

Conclusion ....................................................................................................................... 14
Bibliography ......................................................................................................................... 16
Appendix .............................................................................................................................. 18ii

1 Introduction
The Norwegian Agency for Quality Assurance in Education (NOKUT) runs an annual student survey
about the perceived quality of education in Norwegian bachelor and master programs. The 2014
results were published in February 2015. One of the most striking result is that students, while verysatisfied with the overall quality of their programs, are dissatisfied with the feedback and individual
counselling they receive as students. This is a surprising result considering that feedback andcounselling has been the focus of Norwegian and international education policies for many years.
Around the turn of the century, Norway began a significant reform process of its higher education
system culminating in the “Quality reform” of 2003. The objectives of the reform was to improve thequality of the both education and research, reduce students’ time to completion, and increaseinternational cooperation. To improve the quality of education, the government wanted all higher
education institutions to move away from an evaluation system of final exams and only summativeevaluation, and towards a system of much more formative feedback and evaluation, such as portfolio
evaluation, take home exams and essays. In addition, the government wanted the institutions to focus
more on the individual student through an increased focus on individual advising (Ministry of
Education and Research 2001).
In the more than 10 years that have passed since the reforms were initiated a number of evaluations
have shown that Norwegian higher education institutions still rely heavily on final exams andsummative evaluations (Michelsen and Aamodt 2007). A large number of politicians, student interest
groups, and other interest groups have for years advocated that the higher education institutions needto refocus their methods of evaluation and increase their focus on individual advising, but it seems, tono avail. However, until now, we do not know what students themselves think about the amount andquality of the feedback and advising they receive, and whether student satisfaction with feedback andadvising affects their perception of the overall quality of their study program. We answer thosequestions in this paper.
Based on the national student survey, Studiebarometeret, and interviews with students, faculty, andprogram leaders at seven different study programs in medicine and engineering in Norway, we showthat while students are dissatisfied with the feedback and advising they receive, this dissatisfaction do
not affect the students’ overall perception of the quality of their study programs. We argue that themain reason for this is that students do not expect formative feedback and individual advising to play amajor role in their university education. In other words, when asked about their satisfaction withquantity and quality of the feedback and advising they receive students indicate that they are
dissatisfied, but because they did not expect much—and are largely unaware of the positive effects offeedback and advising—this dissatisfaction do not reduce their satisfaction with the overall quality of
their study program.
In the remainder of this paper, we first discuss some research findings on the positive effects offormative feedback and individual advising. We then discuss our research design and case selection. In
the third section, we present our findings before we conclude with some thoughts about the way
forward.

1

2 BackgroundA significant amount of research shows that formative feedback and individual advising can have apositive effect on student skill and knowledge acquisition, learning, motivation, retention, and overallsatisfaction. In this brief section, we detail some of this research.
The goal of formative feedback is to modify the student’s thinking or behaviour to enhance learning(Shute 2008). According a large body of research show that feedback can “facilitate students’development as independent learners who are able to monitor, evaluate, and regulate their own
learning” (Evans 2013 p. 72; see also Shute 2008; Black and William 2009; Hattie and Timperley2007; Nicol and Macfarlane-Dick 2006). It is important to note that this research also indicates that
not all feedback works. The effects of feedback are varied and research indicates that only certain
types of feedback given under the right conditions is likely to have the desired effects. Even thoughthese are important points we do not discuss it further as the effects of feedback is outside the scope of
this paper.
In addition to enhance learning, feedback can also improve student motivation. Positive feedback can
increase students’ motivation and negate the potentially adverse effects of negative feedback (Lizzio
and Wilson 2008). Other researchers (see for example Narciss and Huth 2004; Wigfield and Eccles
2000; Lepper and Chabay 1985) have also found similar effects. Wigfield and Eccles (2000), argue
that timely and constructive feedback can increase a student’s interest and motivation for problem
solving. Nicol and Macfarlane-Dick (2006) suggests that for formative feedback to have positiveeffects, feedback should encourage, amongst other things, positive motivational beliefs and selfesteem. Even though we have not found any research that explicitly connects student motivation andself-esteem to student retention and completion, the connection seems logical. A student who gains
confidence and motivation in her study program is more likely to complete the program.
Formative feedback is not the only way to enhance student learning, motivation, and overallsatisfaction. Another important factor is student-faculty interaction. A significant number of studies
have shown that student-faculty interaction has a positive effect on student learning, personal
development, and overall satisfaction (Astin 1993; Kuh and Hu 2001; Bjorklund et al; Endo andHarpel 1982; Thompson 2001; Kuh 1995; Pascarella and Terennzini 1980, Tinto 1987). In there muchcited article from 2001, Kuh and Hu argue that increased student-faculty interactions increase theeducational and personal achievement of students, as well as their overall satisfaction with theireducation. Bjorklund et al. (2004) show that student-faculty interactions coupled with constructive
feedback increased engineering students’ problem solving skills, group skills, occupational awareness,and engineering competence. Other studies have shown that the frequency of informal contact between
students and faculty can lead to intellectual growth and student satisfaction (Endo and Harpel 1982).
Pascarella and Ternezini (1980) found that the quality of interactions is more important than thequantity, but that high quality interactions increased students’ intellectual development, as well as,
first-year student persistence. In his 1987 seminal book “Leaving College: Rethinking the Causes andCures of Student Attrition”, Vincent Tinto examines the role of student-faculty interaction and findsthat the quality of student-faculty interaction and the student’s integration into the university are
central factors limiting student attrition (see Hovdhaugen and Aamodt 2005 for a similar argument).
Other research on student satisfaction shows that other factors than feedback and student-facultyinteraction affect how satisfied students are with their study programs and higher education2

institutions. In a study of first-year students in Norway, Wiers-Jenssen et al. (2010) find that student
satisfaction is a function of the social environment, various features of the physical infrastructure andsupport facilities, and the quality of teaching and instruction (189). In their study of student
satisfaction at an U.S. university, Elliot and Healy (2001) finds that student centeredness—that is, thedegree to which the students feel welcome and value—is an important predictor of student
satisfaction. In addition, they find that the campus climate and instructional effectiveness are otherimportant predictors.
In sum, formative feedback and student-faculty interaction (individual advising), are critical factorsthat under the right conditions can increase student learning, motivation, retention and overallsatisfaction. In addition, other factors such as the quality and effectiveness of teaching, the overall
quality of the campus, and the degree to which students feel welcomed and valued explain student
satisfaction. In the next section, we describe our research design and case selection for the qualitative
study.

3 Research Design
In this paper, we answer three separate, but connected research questions. First, why are Norwegianstudents dissatisfied with the feedback and advising they receive? Second, how important is feedback
and advising in students assessment of the overall quality and what further explains the students
overall satisfaction? Finally, why are students satisfied with the overall quality of the program when
they are dissatisfied with important elements of their education (feedback and advising)? To answer
these three questions, we combine insights from the national student survey, Studiebarometeret, withinterview data from seven different study programs in engineering and medicine.
We use a nested analysis design (Liberman 2005), where we first examine the quantitative data to
select cases for the qualitative analysis. We then use insights from the interviews to develop
hypotheses that we test with the data from survey. The mixed-method design allows us to better
answer the questions we pose in this paper. Whereas the survey allows us to say something about
student satisfaction and dissatisfaction, the interviews provides us with a better understanding of what
students value with their study programs and to understand why they are satisfied or dissatisfied withkey aspects of their study programs.
Studiebarometeret is an annual national level student survey run by the Norwegian Agency for Quality
Assurance in Education (NOKUT). NOKUT invites all second year bachelor and master students, aswells as 5th year students in 5-year integrated masters programs and 6-year professional programs toanswer the online survey. The survey consists of approximately 90 questions regarding the students’
perception of the quality of the study program. NOKUT does not ask questions about individual
courses, institutional issues, or student welfare.1 NOKUT asks every student to assess different
aspects of the quality of the study program on a 5-point Likert scale. NOKUT divides the majority of
the answers into seven indexes: (1) learning culture/environment, (2) stimulation and coherence, (3)1 For more information about the survey, see Lid et al (2014) and Damen (2015). See also http://www.nokut.no/en/About-Studiebarometeret/for a detailed description of the survey.

3

working life relevance, (4) teaching and academic counselling, (5) examinations and assignments, (6)learning outcomes, and (7) student influence and participation.
In 2014, the population consisted of 58,000 students and approximately 25,000 students completed thesurvey (42%). Norwegian students are overall very satisfied with the quality of their study programs.
The average score on the statement “I am, all things considered, satisfied with the program I am
currently attending” was 4.1, and 75% of the respondents scored the statement 4 or 5. Despite thestudents’ overall satisfaction, the average scores for the questions “How satisfied are you withfeedback on your work from your teachers?” and “How satisfied are you with individual student
counselling by the teacher?” were only 3.3 and 2.9 respectively. Making them among the lowest
scoring items in the survey. In figure one we display the distribution of the answers to the threequestions.
Figure one: Distribution, whole sample

Figure one illustrates the significant difference in student satisfaction on the three items. Whereas 75
percent of the respondents answer four or five for overall satisfaction, 75 percent of the respondents
answer three or four on satisfaction with feedback, and 75 percent answer two, three or four on
satisfaction with individual counselling. This gap and the lack of a relationship between feedback,counselling, and overall satisfaction is contrary to what the literature suggests that it should be.

3.1 Case selection
We furthered explored these results to identify which study programs to examine in debt. First, wedisaggregated the data by academic fields. The analysis of academic fields showed that students inmedicine, psychology, and engineering had the largest difference in scores between overallsatisfaction and feedback and counselling. We further disaggregated the data by study programs andfrom there we chose 10 study programs to investigate further. Of the 10 programs, seven invited us forinterviews. These study programs were two medical programs (medical doctor programs at the
University of Oslo (UiO) and the Norwegian University of Science and Technology (NTNU)). Two 5year master programs in engineering at NTNU (Product and product development and Materials

4

science and engineering, and three bachelor programs in engineering (Chemical and materials
technology engineering at the Sør-Trøndelag university college (HiST) and Oslo and Akershus
university college (HiOA), and civil engineering at HiOA). These seven programs allowed us to
examine programs at both the masters and bachelor level and at university and university college level.
In addition, the case selection provide us with variation in terms of student satisfaction on all three
indicators. Table one lists the programs and the average scores for each of the three indicators, as wellas, the difference between overall satisfaction and feedback and counselling.
Table 1: Score per program on key indicators
ProgramOverallFeedbackCounselling

Overallfeedback
difference

Overallcounselling
difference

Institution
type

DegreeUiO medicine

4,19

2,25

1,96

-1,94

-2,23

University

Professional

NTNU medicine

4,43

2,7

1,8

-1,73

-2,63

University

Professional

NTNU product

4,04

2,71

2,22

-1,33

-1,82

University

5-yearmasterNTNU material

4,46

3,31

3,3

-1,15

-1,16

University

5-yearmasterHiST chemical andmaterial

4,3

3,42

3,26

-0,88

-1,04

College

Bachelor

HiOA chemical andmaterial

4,5

3,33

2,9

-1,17

-1,6

College

Bachelor

HiOA civil engineering

4,01

2,29

2,15

-1,72

-1,86

College

Bachelor

As we can see from table one, both medicine programs had the largest difference between overallsatisfaction and satisfaction with feedback and counselling. There is also a significant difference
between the two engineering programs at NTNU in satisfaction with feedback and counselling. While
students in product design score relatively low, this is not the case for the students in material
engineering. Students at the two programs in chemical engineering are significantly more satisfied
with feedback and counselling than the civil engineer program at HiOA. Since students score allindicators on a Likert scale, we also show the distribution for each program in box plots below.

5Figure two: Distributions for seven selected programs

Figure two, shows the distribution for all student in the seven programs we chose to interview. The
figure illustrates, not only that students in these programs were much more dissatisfied withcounselling and feedback, than with the overall quality of the program, but also that the students varymuch more in their assessment on feedback and counselling than on overall satisfaction.
We asked each of the seven programs for permission to interview students, faculty and program
leaders. We also asked for help with recruitment of 5-7 students and 2-3 faculty members withteaching obligations. For the professional and 5-year master programs, we interviewed two studentgroups. One consisting of students currently in their 4th semester and one consisting of students
currently in their 10th semester. For the bachelor programs, we interviewed one student-groupconsisting of students currently in their 4th semester. We selected students in their 4th and 10thsemester because only students in those semesters answered the survey in the fall of 2014. There wasno requirement that students had to have answered the survey to participate in the interviews, andindeed, we do not know whether they did or not. The interviews, except for the program leaders, were
all semi-structured group interviews. We recorded all interviews, one of the authors transcribed theinterviews, and all three authors coded and organized each interview. We then created organized asummary of our findings together as a group.
By focusing exclusively on only two academic fields in our qualitative analysis we clearly reduce ourability to generalize based on the findings from our interviews. However, the main purpose of theinterviews is to generate alternative explanations and hypotheses to test with the entire sample of oursurvey. In the end, it is our belief that this back and forth between our quantitative and qualitative data
provided us with the most robust research design.

64 Analysis
In this section, we first discuss why students are dissatisfied with feedback and advising. We thenexplain why students are satisfied with the overall quality of their study program. Finally, we explain
why students’ dissatisfaction with feedback and advising does not affect their assessment of theoverall quality of their study program.

4.1 Feedback
Of the programs we have selected four are university degrees and three are college degrees. Two of
the university programs are 5-year integrated master degrees (engineering) and two are 6-year
professional degrees (medical doctor). The three college degrees are all bachelor degrees inengineering.
All the engineering degrees (bachelor and master) are similarly organized. During the first year
students take introductory classes in several natural science fields (math etc.), then from year two,
student specialize more in their respective fields. Most courses are taught in a lecture format, and fornearly every class throughout the program, students have to turn in weekly or bi-weekly assignments.
These assignments are graded on a pass/fail basis and students must pass a certain number ofassignments to sit for the final exam.2
The medical programs are organized very differently than the engineering programs. At both
universities, the students have a mix of lectures, problem based learning, and practical training at
hospitals and general practitioner clinics. The only form of student assessment is a final exam at theend of each year, and students do not hand in assignments or projects except for a master thesis in theirfifth year.
In general, the students at all seven programs, are relatively dissatisfied with the feedback theyreceive. The mean score on feedback for the entire sample is 3.28 (sd. 1.1), while the mean score forthe programs we interviewed is 2.49 (sd. 1.1). The mean score for the least dissatisfied program is
3.43 (sd. 0.87), while the mean score for the most dissatisfied program is 2.18 (sd. 1.04). Ourinterviews with students indicate the lack of constructive or formative feedback is the most importantsource of the students’ dissatisfaction. When we asked students what kind of feedback they receive,the most common answers were “we only receive a letter grade after our final exam”, or “we don’t
receive any feedback” (or some variation of these two answers). Only after the students discussed thequestion among themselves, did they think of other sources of feedback. The weekly assignments
engineering students, both university and college, hand in, is a potential source for feedback.
However, the student assistants who grade the weekly assignments do not have time to write detailed
feedback. Instead, students only receive a pass or fail on the assignments. All students agreed that the
weekly assignments were valuable in terms of preparation for final exams, but they also agreed thatthe pass/fail notation do not constitute feedback. Indeed, most student do not even pick up the gradedassignments.
Students at the two bachelor programs in chemical engineering was somewhat more satisfied with the
feedback they received compared to students at other programs. The main reason for this was the
2 For most of the classes, the grade on the final exam is the only grade the students receive. For some classes students also complete
individual or group projects. These projects sometimes count towards the student final grade.

7feedback they received on their laboratory reports. Rather than pass/fail, students receive detailed
comments on their lab-reports and are required to revise their reports based on the feedback and turnthem back in for approval. Though time consuming, students found this to be helpful for theirlearning.
The medical students receive no written feedback other than a grade on their annual exams. Students
spend a significant amount of time with faculty and medical doctors, and the potential for oral
feedback is significant. However, students report that most of the interaction takes place in a groupsetting and there is little time for individual feedback.
From our interviews with the students, it seems clear that the main reason for student dissatisfaction
with the feedback they receive relates to the quantity of feedback more than the quality of feedback.
Students indicate that they are satisfied with the feedback they receive on laboratory reports and otherprojects, but in general, they receive a limited amount of formative feedback.
Unfortunately, we do not ask students about the quantity of feedback they receive in their study
program, but we do ask students how much different types of teaching and learning methods are used.
This allows us to statistically test whether different types of teaching methods are more conducive toproviding feedback to students. Figure three is based on an ordinal logistic regression of where
feedback is the dependent variable and different types of teaching methods are the independent
variables (due to space constraints we show the regression table in the appendix).
Figure three: Marginal effects of teaching methods on feedback

The dots represents the predicted probabilities of the logit coefficients. The shaded area represents the 95% confidence
interval.

The figure shows the probability of a student scoring “5” on feedback dependent on a student’s
assessment of how much lecture and written assignments the program uses. The left panel shows thatthe probability of scoring “5” on feedback increases from approximately 11 percent if lectures are
rarely used to 12.3 percent if lectures are frequently used. In other words, lectures, as a teaching8

method does not increase the probability of students being more satisfied with feedback very much.
Indeed as we see from the confidence intervals, the relationship is not statistically significant. The
right panel shows that the probability of scoring “5” on feedback goes from approximately 9 percent
when a program use written assignments sparingly to nearly 14 percent when a program use written
assignments frequently. These results, and other results from the model (see appendix), largely
supports the findings from the interviews.

4.2 Individual advising
Norwegian students are dissatisfied with the individual advising they receive in their programs. This
holds true for the whole sample and for all of the programs we interviewed for this project. For thewhole sample, the mean score on individual counselling is 2.85 (sd. 1.2), while for the programs weinterviewed the mean score on this question was 2.17 (std. dev 1.07). Of the seven programs, the mostsatisfied programs scored 3.3 (sd. 1.4), while the most dissatisfied program scored 1.82 (sd. 0.96).
As with feedback, the main reason—according to the interviewees—for students’ dissatisfaction withindividual advising is the lack of advising and not with the quality. The most satisfied program interms of individual feedback is the 5-year integrated master program in Material Science andEngineering. However, there is a significant difference between the second year students (2.75/sd. 2.0)
and the fourth year students (3.56/sd. 1.0). The second year students are dissatisfied, while the fourthyear students are relatively satisfied. The fourth year students told us that they were quite satisfied
with the advising they receive with their master’s thesis and projects related to their thesis. However,they agreed with the second year students that during the first years of the program they received very
little advising, especially considering the small size of their cohorts.
Students at the bachelor program in chemical engineering at Sør-Trøndelag University College arequite satisfied with the advising they receive. For these students the relatively small size of the
program and the collegiality between students and the teaching staff allow students to meet regularly
with faculty members to discuss assignments, and projects. Students in the program felt that they knew
the teachers and the teachers new them, and this made it easier to seek individual advice whennecessary. Students at the bachelor program in chemistry at Oslo and Akershus University College
largely agreed to these sentiments.
For many of the other programs, the lack of collegiality and familiarity with faculty members is a key
reason for the students’ dissatisfaction with the advising they receive. In several programs, thestudents were frustrated that no faculty members knew their name, and that students could spend five
years in a study program without getting to know any teachers nor would any teachers know them.

As with feedback, we did not ask students about the frequency with which they received individual
counselling. However, based on our interviews it is clear that some teaching methods are more
conducive for individual counselling than others are. To test this statistically we ran an identical model
as with feedback, but changed the dependent variable to individual counselling. Figure four,
illustrates, in even starker terms, how much more conducive, written assignments are to provide
students with individual counselling than lectures.

9Figure four: Marginal effects of teaching methods on individual counselling

The left panel shows that there is no statistically significant relationship between lectures andindividual counselling, whereas the left panel shows that the probability of scoring “5” on individual
counselling increases from approximately 6.5 percent to over 10 percent for a program that use written
assignments frequently rather than rarely. As with feedback, these results, and other results from themodel (see appendix), largely supports the findings from the interviews.

4.3 Overall satisfaction
Norwegian students are generally very satisfied with the overall quality of their study programs. The
average score for the whole sample is 4.05 (sd. 0.98). Students at the seven programs we interviewed
were even more satisfied, with an average score of 4.21 (sd. 0.91). The average score of the mostsatisfied program was 4.54 (sd. 0.68), while the average of the least satisfied program was 4.04 (sd.
0.92). In general, students in all programs were very satisfied with the overall quality and the variation
among the programs is very small.

According to our interviews, the most important factors explaining students’ overall satisfaction are
coherence of the program, relevance of the program, that the program and teaching methods were
stimulating, and the learning and social environment. The majority of the students felt that the
programs were well structured, offered a good mix of courses and assignments and learning methods,
and that these where stimulating. The students also felt that the study programs were very relevant to
the labour market and this was an important factor in their assessment of the overall quality. Finally,
students, especially those studying in Trondheim, emphasized the social environment in their10evaluation of the overall quality of their programs. Many students, especially those studying atcolleges, also emphasized the environment between teachers and students as an important factor.
Interestingly, none of the students we interviewed mentioned feedback and/or individual advising as
an important factor for overall satisfaction. When we asked students to discuss positive and negative
issues related to the program, none of the students referred to feedback or advising as a negative or
positive factor.
In addition to the factors already discussed, it is clear that being admitted to prestigious or very
competitive programs such as medicine or 5-year integrated master degrees at NTNU influence
student satisfaction. Many of the engineering students at NTNU and the medicine students at NTNU
and Oslo prefaced any critique of their programs by pointing out that the programs were Norway’s
best in in its field. As one student said, “When you ask how satisfied I am with the program, then all I
can say is, I’m crazy satisfied! I am going to become a medical doctor, and I was admitted to thisprogram. You will never say that that you are dissatisfied about being a student in this program”(medical doctor student at the UiO). Our statistical analysis shows the same result. Students enrolled
in prestigious fields and schools are more satisfied with the overall quality of their program when we
control for various other factors.
In addition to the interviews, we also examine the results from the student survey statistically. Since
the dependent variable—overall satisfaction with the study program—is measured on a 5-point Likert
scale we run an ordinal logit model.3 Due to space constraints, we present the full model in the
appendix. Here we show the predicted probabilities of the key variables.

3 Since the data is clearly hierarchical — that is, students are enrolled in programs at different universities—we also run multilevel
regression models as robustness checks. The main results of these models are very similar to the ordinal logit models. See the appendix forthese models too.

11

Figure five: Predicted probabilities model 3, overall satisfaction

The figure displays the predicted probabilities of a student answering, “fully agree” to the question: “I
am, all things considered, satisfied with the program I am currently attending”, when each variable inmodel three moves from the minimum value to the maximum value, holding all other variables
constant. In substantive terms, we see from the model that a student is approximately 35 percent more
likely to being very satisfied with the overall quality of her program when she is very satisfied withhow stimulating the teaching is, than when she is very dissatisfied with how stimulating the teaching
is. The model shows that academic stimulation and the coherence of the program has the strongest
effect, while feedback has no effect. Individual counselling does have a small positive effect, but it is
the least important of the quality indicators we ask students to assess. Of the other control variables,
we see that females and older students are less likely to be very satisfied with the overall quality of the
program. Interestingly the program size has no effect on students’ perception of the overall quality ofthe program, and neither does it matter whether students attend a college or university. Finally, we seethat being enrolled in a prestigious program has a small positive effect on students’ overallsatisfaction. These results are remarkably similar to what we found in our interviews of the sevenprograms described above.

4.4 The missing link between feedback, counselling and overall satisfaction
As mentioned above, the students we interviewed did not cite feedback or advising as a contributing
factor to their evaluation of the overall quality of the program. The results from our statistical model
supports this too. Model three in the appendix, shows that feedback has no effect on overallsatisfaction, while individual counselling only has a very small positive effect. Why do students ignore
their dissatisfaction with feedback and counselling when they evaluate the overall quality of their12

study programs, when prior research indicates that feedback and counselling affects overallsatisfaction (see discussion above)?
Prior to our interviews, our working hypothesis was that student dissatisfaction with feedback andadvising was due to their high expectations. In other words, because this generation of students areused to being catered to at the primary and secondary level of schooling, they expect a significant levelof feedback and personal advising at the tertiary level of education as well.4 In order to test thishypothesis we asked students at all seven programs about their expectations to feedback andcounselling prior to enrolling at their programs. The answers were surprising and uniform across all
programs. The students had very low expectations. That is, they did not expect any regular feedback or
counselling. Indeed, they had been warned prior to enrolling in higher education, that at the university
they would be completely responsible for their own learning.
The low expectations students had about feedback and counselling, and the lack of a relationshipbetween student dissatisfaction with feedback and counselling and overall quality makes logical sense.
If high expectations about feedback and counselling explained the low scores, we would expect there
to be a relationship between feedback and counselling, and overall quality. However, if students had
low expectations about feedback and counselling, then they can still be dissatisfied with these aspectsof their program, without the dissatisfaction affecting their assessment of the overall quality of the
program. Several interviewees indicated this to us as well. Multiple students in different programs told
us that though they were dissatisfied with the feedback and counselling they received they were stillsatisfied with the program. Other students told us that feedback and counselling was something they
did not think about until they saw the questions on the survey. A third group of students, in particular
students at NTNU and UiO, seemed to excuse the faculty members for not providing more feedback
and counselling. These students indicated that though they were dissatisfied with the limited feedback
and counselling they received, they accepted this because of the resources it would take to provide
more feedback and counselling.
While our initial hypothesis, that high student expectations affect student satisfaction with feedback
and counselling was incorrect, expectations is still part of the explanation of the missing link between
feedback, counselling, and overall satisfaction. However, rather than high expectations, students do
not expect to receive significant amounts of feedback and counselling at the university level. Neither,does it seem, are students aware of the positive effects feedback and counselling can have on skill andknowledge acquisition, learning, and motivation.5 Thus, when we ask students to assess theirsatisfaction with the feedback and counselling they receive, they are dissatisfied because they receive
limited amounts of both. Yet, this dissatisfaction, because feedback and counselling is relatively
unimportant to the students, do not affect their assessment of the overall quality of their programs.
Our statistical analysis supports this view. If we look at the marginal effects of the variables that mostaffect overall satisfaction and the marginal effects of feedback and counselling we see how feedback
has no positive effect on overall satisfaction, and how counselling only has a very minor effect.

4 Nearly every faculty member and program leader believe that students have high expectations when it comes to feedback and counselling,
and that these expectations affect student assessment. One faculty member referred to students as the “kindergarten” generation to illustrate
this.
5 See Price et al. 2010, and Cotten & Wilson 2006 for similar findings.

13

Figure 6: Marginal effects of feedback, counselling, academic stimulation and learning environment

The figure shows even more clearly than figure five above, how student satisfaction with feedback andcounselling has no (or very little effect) on students’ overall satisfaction. In contrast, we see that theprobability of students being very satisfied with the overall quality of the program rises fromapproximately zero to almost sixty percent when they are very satisfied with academic stimulation andcoherence rather than very dissatisfied.

5 Conclusion
While Norwegian students are dissatisfied with the quantity and quality of the individual feedback andcounselling they receive, they are nonetheless very satisfied with the overall quality of their study
programmes. Considering the importance of individual feedback and counselling on student skill andknowledge acquisition, learning, motivation, retention, and overall satisfaction, we found this to be avery interesting puzzle. In this paper we have shown, through qualitative and quantitative evidence,that students are dissatisfied with feedback and counselling because they receive little of both and thatthe feedback they receive often has limited value to their learning. I.e. it is not very constructive. At
the same time, we found that students do not expect, and seem unaware of the benefits individual
feedback and counselling can have on their learning. These low expectations means that students do
not consider the amount and quality of feedback and counselling as important when they assess theoverall quality of their programmes. Rather, the academic stimulation and the coherence of theirprograms, teachers’ ability to engage the students, the social and academic environment, the perceived
relevance of their education, and the degree to which they are satisfied with what they are learning are
the key factors students use to assess the quality of their programmes.

14

These are interesting and important findings, but should not be used in the wrong way. Norwegian
institutions of higher education cannot, and should not take these results to mean that they do not needto care about feedback and counselling. Students’ lack of expectations and unawareness of the benefitsof feedback and counselling have on skill and knowledge acquisition, learning, and motivation does
not mean that higher education institutions should deprioritize feedback and counselling. Rather, theyshould increase both the quantity and quality of feedback and counselling they provide to theirstudents, while simultaneously make the students aware of the benefits, so that students take fulladvantage of the opportunities the study programmes provide.

15

Bibliography
Astin, A. W. (1993). What matters in college: Four critical years revisited. San Francisco: JosseyBass.
Bakken, Pål et al. (2015). Studiebarometeret 2014: gjennomføring og hovedtendenser,
Studiebarometeret: Rapport 1-2014. NOKUT.
Bjorklund, Stefani A. et al. (2004). Effects of Faculty Interaction and Feedback on Gains in Student
Skills. Journal of Engineering Education, 93(2): 153-160.
Black, Paul and Wiliam, Dylan. (2009). Developing a theory of formative assessment. Educational
Assessment, Evaluation and Accountability, 21:5-31.
Cotton, Sheila R. and Wilson, Bonnie. (2006). Student-faculty interactions: Dynamics anddeterminants. Higher Education, 54(1): 487-519.
Elliott, Kevin M and Healy, Margaret A. (2001). Key Factors Influencing Student Satisfaction Related
to Recruitment and Retention. Journal of Marketing for Higher Education, 10(4): 1-11.
Endo, Jean J. and Harpel, Richard L. (1982). The Effect of Student-Faculty Interaction on Student
Educational Outcomes. Research in Higher Education, 16(2): 115-138.
Evans, Carrol. (2013). Making Sense of Assessment feedback in Higher Education. Review of
Educational Research, 83(1): 70-120.
Hattie, John and Timperley, Helen. (2007). The Power of Feedback. Review of Educational Research,
77(1): 81-112.
Hovdhaugen, Elisabeth and Aamodt, Per Olaf. (2005). Frafall fra Universitetet. Arbeidsnotat 13/2005.
NIFU STEP Norsk institutt for studier av forskning og utdanning / Senter for innonvasjonsforskning.
King, Gary, Michael Tomz, and Jason Wittenberg. (2000). Making the Most of Statistical Analysis:
Improving Interpretation and Presentation. American Journal of Political Science 44 (2): 347–61.
Price, Margaret et al. 2010. Feedback: all that effort, but what is the effect. Assessment and Evaluationin Higher Education, 35(3). 277-289.
Kuh, George. D. (1995). The other curriculum: Out-of-class experiences associated with student
learning and personal development. The Journal of Higher Education, 66(2), 123-155.
Kuh, George D. and Hu, Shouping. (2001). The Effects of Student-Faculty Interaction In the 1990s.
The Review of higher Education, 24(3): 309-332
Lid, Stein Erik et al. (2014). Student views on quality in their study programs – what matters?
Studiebarometeret: Rapport 12-2014. NOKUT.
Lieberman, Evan S. (2005). Nested Analysis as a Mixed-Method Strategy for Comparative Research.
American Political Science Review, 99(3): 435-452.

16

Lizzio, Alf and Wilson, Keithia. (2008). Feedback on assessment: students’ perceptions on quality andeffectiveness. Assessment & Evaluation in Higher Education, 33(3): 263-275.
Lepper, Mark A. and Chabay, Ruth W. (1985). Intrinsic Motivation and Instruction: Conflicting Views
on the Role of Motivational Processes in Computer-based Education. Educational Psychology. 20(4):217-230.
Michelsen, Svein and Aamodt, Per Olaf. (2007). Evaluering av Kvalitetsreformen: Slutt rapport.
Norwegian Research Council.
Ministry of Education and Research. (2001). Gjør din plikt – Krev din rett; Kvalitetsreform av høyere
utdanning. Government white paper nr. 27.
Narciss, S. & Huth, K. (2004). How to design informative tutoring feedback for multimedialearning.
In H. M. Niegemann, D. Leutner & R. Brunken (Eds.), Instructional design for multimedia
learning (s. 181–195). Munster, New York: Waxmann.
Nicol, David J. and Macfarlane-Dick, Debra. (2006). Formative assessment and self-regulated
learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2):199-218.
Pascarella, E. T., & Terenzini, P. T. (1979). Student-faculty informal contact and college persistence:
A further investigation. Journal of Educational Research, 72: 214-218.
Shute, Valerie J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1): 153189.
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition(2d ed.). Chicago: University of Chicago Press.
Thompson, M. D. (2001). Informal student-faculty interaction: Its relationship to educational gains in
Science and Mathematics among community college students. Community College Review, 29(1), 3557.
Wiers-Jenssen, Jannecke et al. (2002). Student Satisfaction: Towards an empirical deconstruction of
the concept. Quality in Higher Education, 8(2): 183-195.
Wigfield, A. & Eccles, J. S. (2000). Expectancy-value theory of achievement
motivation. Contemporary Educational Psychology, 25(1), 68–81

17Appendix
In this appendix, we present the full regression models discussed in the paper, as well as the multilevel
regression models we have run as robustness checks. First, we describe the variables we use in all themodels. We then present the models.
Data
In model one we use student answers to the question “How satisfied are you with feedback on yourwork given by the teachers (is the feedback constructive)” as the dependent variable. Students
answered on a 5-point Likert scale from “not satisfied” (1) to “very satisfied” (5). As the principle
independent variables, we use student answers about the frequency of different teaching and learning
methods in their programs. The question was “To what degree are these teaching and learning methods
used in your programme?” Here to the response alternatives were on a 5-point Likert scale, from “to alow degree” (1) “to a high degree” (5). The students could also answer, “not used”. We transformed
the variable to a six-point Likert scale were “not used” equals zero.
In model two we use student answers to the question “How satisfied are you with individual student
counselling given by the teachers» as the dependent variable. The response alternatives and principle
independent variables were the same as for model one.
As control variables, we included both individual level variables, and program and institution
variables. We controlled for gender, age and study progression. Study progression is the average
number of credit hours a student has taken during the last three semesters. At the program level, we
control for whether students are bachelor students, second year master students, or fifth year masterstudents in a 5-year integrated master or professional program. To control for the size of the program
we used the total number of students in each program that received the survey. Finally, we controlled
for whether the program was a prestigious program or not. In the list of prestigious programs, we
included the four medical programs in Norway, all 5-year integrated engineering programs at the
Norwegian University of Science and Technology, all professional psychology degree programs, andall professional law degree programs. At the institutional level we control for whether the institution
is a university of not, and the size of the institution in terms of the total number of students.
In model three, we use student answers to the question “To what degree to you agree that, I am, allthings considered, satisfied with the programme I am currently attending?” Here too students
answered on a 5-point Likert scale from “do not agree” (1) to “fully agree” (5). In this model theprinciple independent variables are student responses to questions regarding the quality of the
program. We divided the questions into seven indexes, which we used as the independent variables.
We use the same control variables in model three as in model one and two.

18

Model one and two: Ordinal logistic regressions. Feedback and counselling
Model 1Model 2
Satisfaction with feedback
Satisfaction with counselling
B
SE*
B
SE*
USE OF TEACHING METHODS
Lectures
0,03
0,02
-0,03
0,02
Seminar
0,11
0,01
0,11
0,01
Group work without teacher
0,02
0,01
0,03
0,01
Written assignments
0,13
0,01
0,12
0,02
Projects
0,01
0,01
0,05
0,01
Field work/data collection
0,03
0,01
0,05
0,01
Laboratory
-0,04
0,01
-0,02
0,01
Other practical work
0,08
0,01
0,12
0,01
Case
0,03
0,01
0,02
0,01
Simulation/role play
0,00
0,01
0,02
0,01
Practice training
-0,02
0,01
-0,03
0,01
Digital work methods
0,06
0,01
0,08
0,01
INDIVIDUAL BACKGROUND
FemaleAgeStudy progression

0,00
0,01
-0,00

0,03
0,00
0,00

PROGRAM BACKGROUND
Bachelor-0,15
0,06
Master 2nd year
0,06
0,07
th
Master 5 year
-0,28
0,09
Program size
-0,00
0,00
Prestigious program
-0,30
0,10
University
0,13
0,05
Institution size
-0,00
0,00
N
18611
Clusters
1584* Robust standard errors, clustered on study program.
Bold coefficients are statistically significant at the 95% level.

-0,09
0,02
-0,00

0,03
0,00
0,00

-0,14
0,30
0,04
-0,00
-0,37
-0,01
-0,00
18643
1585

0,07
0,08
0,10
0,00
0,12
0,05
0,00

19

Model three: Ordinal logistic regression. Overall satisfaction
Model 3Overall satisfaction
B

SE*

INDEXES
Teaching and counselling:
Stimulating teaching
Teachers providing good explanations
Teaching covering the curriculum
Feedback
Individual counselling
Index learning environment
Index influence and participation
Index stimulation and coherenceIndex relevanceIndex assessment methods
Index learning goals

0,41
0,17
0,22
0,00
0,04
0,36
0,31
1,06
0,41
-0,06
0,62

0,02
0,02
0,02
0,01
0,01
0,03
0,02
0,03
0,02
0,02
0,03

Individual backgroundFemaleAgeStudy progression

-0,16
-0,00
0,00

0,03
0,00
0,00

Program background
Bachelor
0,19
Master 2nd year
0,06th
Master 5 year
0,01
Program size
0,00
Prestigious program
0,27
University
0,06
Institution size
0,00
N
17842
Clusters
1569* Robust standard errors, clustered on study program.
Bold coefficients are statistically significant at the 95% level.

0,06
0,07
1,00
0,00
0,09
0,06
0,00

Multilevel regression analysis
Since we deal with data collected in a hierarchical nested structure (students within study programs
within institutions), meaning that we do not have completely independent cases at the individual level.
When performing regression analyses, it is necessary to take the clustering of individuals into account.
Ordinary linear regression analysis (OLS) assumes that individual cases are completely independent
from each other. If this is not the case (as in our data), you overestimate the standard deviations, which
can lead to a wrong decision on the significance of effects. We therefore performed multilevel
regression analysis, which takes into account the nested structure of the data.

20

We discerned background characteristics at three ‘levels’ of analysis. Some characteristics represent
the level of institutions: type and size of institutions. Other characteristics represent the program level:
bachelor-master, program size, whether the program is prestigious or not. The third category ofbackground characteristics represent the individual level (gender, age, and study progression).
The satisfaction indexes (overall satisfaction) and teaching methods (satisfaction with feedback andwith individual counselling) function as explanatory variables, in order to explain the overallsatisfaction of students and their satisfaction with feedback and individual counselling.
Multilevel modelling starts with a zero-model, without any explanatory variables. The basic modelshows the unexplained variance at the different levels of analysis. The unexplained variance is largest
at the individual level and smallest at the institutional level. This means that individual characteristics
are more influential in explaining individual differences in satisfaction than institutional or program
characteristics are.
In order to facilitate the interpretation of the effect sizes, we standardized the dependent variables(overall satisfaction, satisfaction with feedback and with individual counselling) to a scale from 1 to
100 (percentile scores). The effect sizes now show how many points on a scale from 1 to 100 thesatisfaction scores increases or decreases under influence of the specific variable, controlled for allother variables in the model.
The results of the multilevel models shown below are highly comparable to the results of the ordinal
logit models in the paper, which strengthen our conclusions.

21

Multilevel regression analysis: dependent variables feedback & individual counselling
FEEDBACK (1-100)
Model 0B

INDIVIDUAL COUNSELLING (1-100)

Model 1SE

Model 0B

SE

Lecture0.669

Seminar

1.867

B

Model 1
SE

B

SE

0.270

-0.79

0.262

0.155

1.916

0.151
0.146

USE OF TEACHING METHODS (0-5)

Group without teacher

0.561

0.152

0.753

Written assignments

2.654

0.202

1.731

0.196
0.161

Project work

0.807

0.166

1.181

Field work/data collection

0.855

0.159

1.163

0.154

Laboratory work

-0.349

0.165

0.013

0.162
0.147

Other practical work

1.258

0.152

1.538

Case

0.611

0.158

0.576

0.154

Simulation/role play

0.166

0.197

0.457

0.191
0.165
0.133

Practice training

0.074

0.167

-0.093

Digital (electronic) work methods

0.985

0.138

1.109

INDIVIDUAL BACKGROUND
women (ref)

0

0

men

0.666

0.432

1.976

0.416

age

0.177

0.036

0.218

0.035

study progression-0.007

0.028

0.027

0.027

PROGRAM BACKGROUND
bachelor (ref)00master 2nd year

3.799

0.798

8.554

0.823

master/professional study 5th year

1.345

1.175

7.404

program size

-0.0160.005-0.029

1.174
0.005

prestigious program-10.046

1.945-11.925

2.049

INSTITUTIONAL BACKGROUND
new university (ref)0university college-1.476

specialized university
other university< 350 (ref)2.480-0.674

2.838

0.268

2.794

-0.241

3.159

0.557

2.296

0.352

2.624

0

0

< 950

-4.907

1.787-5.5221.928< 2000-8.9901.925-10.616

2.113> 2000-9.9282.257-13.2262.505

25.804

3.363

intercept

22056.7010.997

28.376

3.627

3.075

54.86012.82412.5004.906

130.2797.329

77.397

3.991
5.449

6.904

641.0166.150

576.150

6.343

56.50

0.775

Institutional level (N=58)

52.77

7.190

8.905

Program level (N=1738)

127.435.748

61.463

Individual level673.396.506

626.034

N students

23767

17749

23261

17872

-2* log likelihood

224123.8

165345.7

218284.8

165145.0

Multilevel regression analysis: dependent variable overall satisfaction (scale 1-100)
Model 0B

Model 1SE

B

SE

Stimulating teachers

3.701

0.241

Good explaining teachers

1.578

0.238

Teaching covers curriculum

1.576

0.214

Feedback on your work by teachers

0.004

0.188

Individual student counselling

0.675

0.184

Learning environment

3.696

0.280

Influence and participation

2.699

0.222

Academic stimulation and coherence

9.852

0.314

Working life relevance

4.092

0.271

Student assessment-0.593

0.273

Learning goals

5.430

0.354

INDEXES
Teaching and counselling:

INDIVIDUAL BACKGROUND
women (ref)

0

men

1.126

0.331

age

-0.0070.028

study progression

0.060

0.021

PROGRAM BACKGROUND
bachelor (ref)0master 2nd year

-1.6200.506

master/professional study 5th year

-2.541

0.806

program size

0.004

0.003

prestigious program (0-1)

2.543

1.074

INSTITUTIONAL BACKGROUND
new university (ref)0university college-0.432

0.915

specialized university

1.426

1.149

other university

2.801

0.852

< 350 (ref)

0

< 950

0.330

0.953

< 2000-1.4650.953> 2000-1.8871.031-75.210

1.974

intercept

51.6130.768

Institutional level (N=58)

28.990

7.211

0.427

0.420

Program level (N=1738)

69.8394.878

11.944

1.706

Individual level

651.4476.281

372.557

4.191

N students

22927

16847

-2* log likelihood

214943.1

147969.2

23