# Exploring the Relationship between the Collegiate Learning Assessment, Student Learning Activities, and Study Behaviors: Implications for Colleges and Universities

Authors: Theodore Kaniuka; Matthew Wynne
Journal: Education Research International
 (2019)

---

## Abstract

Globally, institutions of higher learning have attempted to utilize various methods to assess student learning outcomes and simultaneously determine what factors influence targeted performance measures. The National Survey of Student Engagement is a popular instrument many colleges and universities employ to gain an understanding of what student behaviors are linked to such desired outcomes as graduation and persistence. Internationally, other universities are exploring the concept of student engagement as a means to assess college environments. Recently, calls from stakeholders have prompted institutions to use assessments to demonstrate so-called twenty-first century skills such as critical thinking; the Collegiate Learning Assessment is one such tool. This study reports preliminary efforts to link the NSSE engagement indicators to CLA performance at a medium-sized Historically Black College or University. Results indicate that the NSSE indicators are (1) poor predictors of student GPA, (2) of the ten indicators of the NSSE, only one was found to be a significant predictor of CLA performance, and (3) of the items comprising this indicator, only half are associated with CLA outcomes.


---

## Body

## 1. Introduction

Recent events in defining the role and purpose of higher education from the federal and local levels have been the impetus for many institutions of higher learning to explore ways to demonstrate that a postsecondary degree matters [1]. These pressures are present worldwide as the role of a college education has shifted from that of the private to public well [2, 3], such that colleges and universities have aligned their focus to service business and industry in order to remain economically viable and receive the support of both the individual and the society [4, 5]. As a result of these pressures, institutions of higher learning have been tasked with teaching and assessing the degree to which a college degree can improve attributes such as the ability to think logically and solve problem [6, 7]. In an effort to demonstrate the valued-added nature of the college degree, these institutions have expanded accountability and assessment initiatives to demonstrate to stakeholders that college matters.There have been multiple accountability initiatives in higher education [8] that have called for the direct assessment of student learning in ways that provide comparable information across institutions and states [9, 10]. Using regression analysis and archival data, this article examines to what degree measures of student learning and engagement experiences are related in an effort to identify the link between learning activities and performance to support postcollege career readiness is possible [11]. These researchers state that, despite the numerous examples of institutions using the National Survey of Student Engagement [12] to monitor programs that are linked to enhancing student engagement, there is a paucity of studies that attempt to link student engagement to desired student outcomes. This study does so by utilizing regression analysis to explore if there are links between student performance on the Collegiate Learning Assessment [13], student learning activities, and study skills as reported by the National Study of Student Engagement in an effort to determine if such a process could be useful in providing insight into how colleges and universities may better align the engagement work they do with the aims and outcome-associated measures of student performance.


## 2. Literature Review

In the twenty-first century, the skills essential for today’s students are far different from those required in the previous decades. A greater emphasis is being given to college readiness and college outcomes because employers now more than ever expect college graduates to possess writing, critical thinking, and problem-solving skills [14] in response to the changing demands of available jobs [15, 16]. Teachers and students can no longer rely solely on the accumulation of disciplinary knowledge and skills as employers evaluate potential employees not purely on what they know, but also on what they can do and how they do it.There is now an emphasis and precedent on focusing on the twenty-first century skills [17, 18] (SBAC, 2012) in addition to knowledge in specific content domains [19–22] in hopes of fostering the development of critical thinking, problem-solving, communication, collaboration, creativity, and innovation skills [19], skills coveted by employers. The impetus to change instructional programs is multifaceted such that it includes various stakeholder groups including prospective students, the very customer colleges, and universities trying to attract. In a recent annual survey conducted by UCLA, Wyer [23] reports that, for the first time, freshman state that getting a better job is most important to attend college.


### 2.1. Assessing Learning and Engagement

Assessment in higher education is a complex issue not only with regard to its purpose but also includes social justice perspectives, equality, and overall public policy considerations of what is assessed in higher education [24]. Messick suggests that assessment for learning in higher education must serve two groups: (1) the faculty must get feedback to support changes in instruction to enhance student outcomes and (2) students must learn from assessment so they can reflect on their own learning practices to better support their knowledge and skill development. Furthermore, it is posited that administrators as well as key policy-makers must also be aware of how the types of learning activities instructors utilize are linked to these same immediate pedagogical outcomes, which includes the long-term implications the college experience has for students and society [25–29]. Assessment is a multifaceted activity often intended to serve many related and divergent purposes. An institution must be clear on what assessment could be added not only in terms of measuring student outcomes but also how these outcomes can be influenced by the actions of faculty and administrators.


### 2.2. Collegiate Learning Assessment

The Collegiate Learning Assessment (CLA) is designed to assess the skills most critical for success beyond college [30, 31]. CLA provides a reliable measure of students’ ability to solve complex problems by analyzing and synthesizing disparate pieces of information, while presenting cohesive, persuasive, and grammatically correct solutions [32, 33], and its values to do so have received international attention and application [34]. The research on the CLA+ as a predictive measure of multiple postcollege outcomes was presented in [33, 35]. The CLA+ was found to gauge and predict factors such as employment, salary, and graduate school attendance. These findings support the utilization of such assessments by employers and institutions of higher learning as a method to determine if applicants and graduates are suitable for employment and further education [36]. It has been suggested that the CLA+ can among other things provide information to potential employers, graduate schools, and communicate achievement standards that can serve to certify the rigor of a school’s educational programs [33, 35, 37].


### 2.3. Student Engagement

Assessing student engagement has become a critical activity for institutions of higher learning globally. Recent efforts in the United Kingdom [38] illustrate the growing attention student engagement is acquiring. In the United States, the National Study of Student Engagement (NSSE) has been utilized by institutions of higher learning to better understand how to measure student engagement and then to assess the degree to which these schools provide their students with value-added learning environments [39, 40]. It also has been argued that the survey can support the actions for faculty to better align their behaviors to better achieve certain desired outcomes for students [41]. Kinzie et al. [42] comment that the NSSE can be a useful tool in monitoring institutional practices and student behaviors that have been linked to certain outcomes. However, according to Gordon et al. [11], schools that use the NSSE should not be concerned on how well they do on the NSSE, but rather should focus on the application of the results to develop a better understanding of how they can improve student outcomes by linking the NSSE to actual instructional activities, and these pedagogical actions support school success [43]. The use of the NSSE data for institutional purposes, specifically for improvement student outcomes, has been criticized with regard to the reliability and validity of the results [44]. However, as these criticisms are not without merit, as Pike [45] previously stated that these shortcomings necessarily need to be evaluated within context and that it is important to apply any set of data carefully and understand the intended purpose of the data. Interestingly, Pike [44] posits that the results from the NSSE are (1) dependable measures of student engagement in educational practices when at least 50 students are used and (2) the NSSE institutional benchmark scores were significantly related to graduation and retention. He further provides that it is possible the use these scores to gauge the engagement of students and evaluate the effectiveness of institutional actions and programs to improve student engagement and academic success.Despite the robust research on student engagement and the relationship to student learning [46–48], supporting student retention [49, 50], and impacting student persistence [51], faculty and administrators still struggle to effectively assess student engagement at lower levels within the college or university. Considering the preceding arguments, it is argued that analyzing the relationship between the NSSE and the CLA is consistent with the recent emphasis on predicting the postcollege activities of graduates as being important for both college officials and potential students. Furthermore, the potential results from this study may yield powerful policy tools to institutional leaders, faculty, and students as to select which institutions of higher learning offer the most value [52].


### 2.4. Research Questions

The following questions were used to guide the study by examining the possible relationships between student engagement and critical thinking as defined above:(1)
Are the NSSE engagement indicators associated with students’ GPA?(2)
Are specific items in the NSSE related to the performance scores on the CLA?


#### 2.4.1. Research Question 1

Finding that benchmark scores of the NSSE are related to institutional retention and graduation rates [11, 44] may suggest that the scores from the benchmarks can serve as measures of the influence of institutional programs and practices that are intended to enhance student success. While previous research has demonstrated weak relationships between NSSE benchmark scores and certain student outcomes [53], additional exploration is warranted.


#### 2.4.2. Research Question 2

This question explores the possible relationships of the student engagement measures used in the NSSE with the critical thinking outcomes of the CLA. The research of Niu et al. [54] showed that there are limited effects of the interventions colleges and universities have implemented in an effort to affect the critical thinking skills of students. Examining how these lower level measures are related to measures of student performance has shown that, although the macromeasures conceptualized and the benchmarks are useful, it is not without limitations, and looking at the microlevel measures may yield relationships not revealed by using aggregated methods [11].


## 2.1. Assessing Learning and Engagement

Assessment in higher education is a complex issue not only with regard to its purpose but also includes social justice perspectives, equality, and overall public policy considerations of what is assessed in higher education [24]. Messick suggests that assessment for learning in higher education must serve two groups: (1) the faculty must get feedback to support changes in instruction to enhance student outcomes and (2) students must learn from assessment so they can reflect on their own learning practices to better support their knowledge and skill development. Furthermore, it is posited that administrators as well as key policy-makers must also be aware of how the types of learning activities instructors utilize are linked to these same immediate pedagogical outcomes, which includes the long-term implications the college experience has for students and society [25–29]. Assessment is a multifaceted activity often intended to serve many related and divergent purposes. An institution must be clear on what assessment could be added not only in terms of measuring student outcomes but also how these outcomes can be influenced by the actions of faculty and administrators.


## 2.2. Collegiate Learning Assessment

The Collegiate Learning Assessment (CLA) is designed to assess the skills most critical for success beyond college [30, 31]. CLA provides a reliable measure of students’ ability to solve complex problems by analyzing and synthesizing disparate pieces of information, while presenting cohesive, persuasive, and grammatically correct solutions [32, 33], and its values to do so have received international attention and application [34]. The research on the CLA+ as a predictive measure of multiple postcollege outcomes was presented in [33, 35]. The CLA+ was found to gauge and predict factors such as employment, salary, and graduate school attendance. These findings support the utilization of such assessments by employers and institutions of higher learning as a method to determine if applicants and graduates are suitable for employment and further education [36]. It has been suggested that the CLA+ can among other things provide information to potential employers, graduate schools, and communicate achievement standards that can serve to certify the rigor of a school’s educational programs [33, 35, 37].


## 2.3. Student Engagement

Assessing student engagement has become a critical activity for institutions of higher learning globally. Recent efforts in the United Kingdom [38] illustrate the growing attention student engagement is acquiring. In the United States, the National Study of Student Engagement (NSSE) has been utilized by institutions of higher learning to better understand how to measure student engagement and then to assess the degree to which these schools provide their students with value-added learning environments [39, 40]. It also has been argued that the survey can support the actions for faculty to better align their behaviors to better achieve certain desired outcomes for students [41]. Kinzie et al. [42] comment that the NSSE can be a useful tool in monitoring institutional practices and student behaviors that have been linked to certain outcomes. However, according to Gordon et al. [11], schools that use the NSSE should not be concerned on how well they do on the NSSE, but rather should focus on the application of the results to develop a better understanding of how they can improve student outcomes by linking the NSSE to actual instructional activities, and these pedagogical actions support school success [43]. The use of the NSSE data for institutional purposes, specifically for improvement student outcomes, has been criticized with regard to the reliability and validity of the results [44]. However, as these criticisms are not without merit, as Pike [45] previously stated that these shortcomings necessarily need to be evaluated within context and that it is important to apply any set of data carefully and understand the intended purpose of the data. Interestingly, Pike [44] posits that the results from the NSSE are (1) dependable measures of student engagement in educational practices when at least 50 students are used and (2) the NSSE institutional benchmark scores were significantly related to graduation and retention. He further provides that it is possible the use these scores to gauge the engagement of students and evaluate the effectiveness of institutional actions and programs to improve student engagement and academic success.Despite the robust research on student engagement and the relationship to student learning [46–48], supporting student retention [49, 50], and impacting student persistence [51], faculty and administrators still struggle to effectively assess student engagement at lower levels within the college or university. Considering the preceding arguments, it is argued that analyzing the relationship between the NSSE and the CLA is consistent with the recent emphasis on predicting the postcollege activities of graduates as being important for both college officials and potential students. Furthermore, the potential results from this study may yield powerful policy tools to institutional leaders, faculty, and students as to select which institutions of higher learning offer the most value [52].


## 2.4. Research Questions

The following questions were used to guide the study by examining the possible relationships between student engagement and critical thinking as defined above:(1)
Are the NSSE engagement indicators associated with students’ GPA?(2)
Are specific items in the NSSE related to the performance scores on the CLA?


### 2.4.1. Research Question 1

Finding that benchmark scores of the NSSE are related to institutional retention and graduation rates [11, 44] may suggest that the scores from the benchmarks can serve as measures of the influence of institutional programs and practices that are intended to enhance student success. While previous research has demonstrated weak relationships between NSSE benchmark scores and certain student outcomes [53], additional exploration is warranted.


### 2.4.2. Research Question 2

This question explores the possible relationships of the student engagement measures used in the NSSE with the critical thinking outcomes of the CLA. The research of Niu et al. [54] showed that there are limited effects of the interventions colleges and universities have implemented in an effort to affect the critical thinking skills of students. Examining how these lower level measures are related to measures of student performance has shown that, although the macromeasures conceptualized and the benchmarks are useful, it is not without limitations, and looking at the microlevel measures may yield relationships not revealed by using aggregated methods [11].


## 2.4.1. Research Question 1

Finding that benchmark scores of the NSSE are related to institutional retention and graduation rates [11, 44] may suggest that the scores from the benchmarks can serve as measures of the influence of institutional programs and practices that are intended to enhance student success. While previous research has demonstrated weak relationships between NSSE benchmark scores and certain student outcomes [53], additional exploration is warranted.


## 2.4.2. Research Question 2

This question explores the possible relationships of the student engagement measures used in the NSSE with the critical thinking outcomes of the CLA. The research of Niu et al. [54] showed that there are limited effects of the interventions colleges and universities have implemented in an effort to affect the critical thinking skills of students. Examining how these lower level measures are related to measures of student performance has shown that, although the macromeasures conceptualized and the benchmarks are useful, it is not without limitations, and looking at the microlevel measures may yield relationships not revealed by using aggregated methods [11].


## 3. Methods

This study uses multivariate linear regression using maximum likelihood estimation with missing values utilizing robust standard errors. To accomplish this, extant data from several past administrations of both the NSSE and CLA were retrieved via institutional research. The participants were freshmen and senior students from 2013 to 2015 that took both the NSSE and CLA. Students are not required to complete the NSSE; therefore, the engagement indicators are weighted to improve the representativeness of the sample to the larger student population. In contrast, the CLA is given to all students as a freshman and again as a senior to measure the value-added experiences of attending the university; therefore, no weighting is required. Table1 reports the demographic data on the sample.Table 1
Select demographic variables for freshmen and senior students.
Variable
Freshmen
Seniors
N
Mean
SD
N
Mean
SD
GPA
428
2.65
0.71
315
3.26
0.54
N
Percent
N
Percent
Race
American Indian
2
0.4
9
2.87
Asian
2
0.4
NA
Black
351
82.24
171
54.46
Hispanic
17
3.97
19
6.1
White
23
5.38
64
20.38
Multiracial
34
7.81
42
13.5
Sex
Female
289
67.52
229
72.93
Not disclosed
5
1.13
9
2.86
Parent education
High school
100
27.24
82
30.94
Attended some college
59
16.1
40
15.09
Associates
71
19.3
43
16.22
Bachelorette
61
16.62
42
15.84
Masters
48
13.1
22
8.3
Doctorate
6
1.63
3
1.13In Table2 the descriptive statistics for the aggregated level of the regression variables are reported. To investigate the relationship between measures associated with student engagement and problem-solving skills, this study replicated to some degree the work of Gordon et al. [11] such that both the aggregate measures of student engagement as defined on the NSSE were regressed onto CLA performance and then the individual questions which are grouped by these identifiers were also regressed on the same outcome measures. To establish parsimonious models the demographic variables in Table 1 were originally included in the regression analysis, and it was found that these factors did not significantly account for differences in either the engagement indicators of CLA scores to the extent that including them in the final models reduced the power and failed to improve the overall precision.Table 2
Descriptive statistics for aggregate regression variables.
Variable
N
M
SD
All
Freshmen
Seniors
All
Freshmen
Seniors
All
Freshmen
Seniors
Engagement indicators
Higher order
668
383
285
40.77
38.78
43.33
14.13
13.61
14.36
Reflective and integrative
709
408
301
37.28
34.73
40.89
13.29
12.92
12.95
Learning strategies
650
372
278
42.67
40.04
46.06
13.88
14.1
12.83
Quantitative reasoning
694
401
293
32.26
31.53
33.23
16.62
15.95
17.48
Collaborative
696
402
294
32.17
33.42
30.58
14.51
13.63
15.51
Discussions with diverse others
665
382
283
40.06
36.99
43.89
17.32
16.44
17.29
Student-faculty interactions
695
401
294
28.56
30.0
26.89
15.74
15.32
16.13
Effective teaching
711
409
302
41.81
40.01
44.25
14.45
14.19
14.47
Quality of interactions
654
383
271
42.25
40.65
44.04
12.23
11.89
12.49
Supportive environment
654
373
260
38.69
39.61
36.68
14.24
14.2
14.14
CLA
Total score
708
408
300
979.08
931.6
1043.65
136.5
119.94
131.19
GPA
314
91
223
3.16
2.97
3.23
0.55
0.51
0.54
Note: highest score for NSSE engagement indicator: 60; CLA: 1515.00.


## 4. Results

The first analysis was to regress the NSSE engagement indicators onto the student cumulative GPA at the time of taking the NSSE to address research question 1. The results shown in Table3 indicate that there is little predictive power of the engagement indicators and institution research-reported GPA as of the ten engagement indicators, only four were found to have coefficients different from zero. They were as follows: Student-Faculty Interaction, Supportive Environment, Quality Interaction, and Effective Teaching for all students. When student class was used to create groups, two engagement indicators were found to have some predictive power for freshman status of students—Student-Faculty Interactions and Effective Teaching. This is in stark contrast for senior class students as the predictive ability of the NSSE engagement indicators of student GPA revealed that not one of the indicators was found to have an estimated coefficient different from zero.Table 3
Regression results for GPA and NSSE engagement indicators with robust standard errors.
NSSE indicator
All
Freshmen
Seniors
B
SE
z
B
SE
z
B
SE
z
Higher order
0.002
0.003
0.83
0.002
0.004
0.64
<−0.001
0.003
−0.03
Reflective
0.004
0.003
1.54
−0.002
0.004
−0.56
−0.002
0.003
−0.47
Learning strategies
0.004
0.002
1.83
<0.001
0.003
0.04
0.003
0.003
1.06
Quantitative reasoning
−0.001
0.002
−0.53
<−0.001
0.003
−0.06
<−0.001
0.002
−0.14
Collaborative learning
−0.002
0.003
−1.13
0.006
0.003
1.87
−0.004
0.003
−1.63
Diverse others
0.001
0.002
0.62
−0.003
0.003
−1.09
0.002
0.002
1.09
Student faculty
−0.008∗∗∗
0.002
−3.52
−0.008∗∗∗
0.003
−2.76
<−0.001
0.003
−0.22
Effective teaching
0.008∗∗∗
0.002
3.6
0.007∗∗
0.003
2.42
0.005
0.003
1.77
Quality interactions
0.005∗∗
0.003
2.05
0.004
0.004
1.01
<0.001
0.003
0.18
Supportive environment
−0.005∗∗
0.002
−2.14
<0.001
0.003
0.21
<−0.001
0.003
−0.26
Constant
2.41∗∗∗
0.12
19.67
2.34∗∗∗
0.16
14.64
3.01∗∗∗
0.13
18.69
Note:∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001; all estimated constants are significant at p<0.001.For the second question, a regression was run to determine if the NSSE engagement indicators are predictive of CLA performance. Prior to running this regression, a simple pairwise correlation was run between GPA and overall performance on the CLA, and it was found there was a significant positive relationship (r = 0.4, p<0.001, r2 = 0.16); a small correlation exists between the two. Therefore, it was expected that if a relationship between NSSE and CLA was to be discovered, it would be limited. As shown in Table 4, using all students, of the ten indicators, four were found to have coefficients different from zero, Student-Faculty Interactions, Diverse Discussion (DD), Collaborative Learning (CL), and Reflective and Integrated Learning (RI). The predicted coefficients had both negative and positive values and ranged in size with Collaborative Learning having the largest negative coefficient implying that a one-unit change in this value was predicted to lower a CLA score by approximately 2.3 points or less than 1%. Both Diversity and Reflective and Integrated Learning had positive coefficients with RI having been found to increase CLA scores by 2.09 points for every one-unit change, a very small percentage increase. When examining these relationships by student class status, there is a simple difference on how each class of students perceives engagement being a factor on CLA performance sharing only one indicator. For freshman, RI, CL, and Learning Strategies (LS), all have estimated coefficients different from zero. While for senior students, only DD and CL have nonzero estimated coefficients. A test was run to determine if the difference in estimators was significant using(1)z=B1−B2seB12+seB22,with z=0.19, indicating that the difference in estimators was not significant. The sign for CL was negative for both groups indicating that as either group reported more frequent occasions of being involved in collaborative settings, overall CLA scores were predicted to decrease. For freshman, RI and LS appear to predict CLA scores positively, such that for freshmen who reported high frequencies of each behavior, their CLA scores were predicted to increase albeit in small amounts. For example, a one-unit increase of RI was estimated to increase CLA scores by 1.48 or well less than a 1% change. Seniors reported that working with and being exposed to others different from themselves tend to increase CLA scores, again small in magnitude. The small magnitude of change in CLA scores for seniors was similar to what was found with freshmen.Table 4
Regression results for CLA total score and NSSE engagement indicators with robust standard errors.
NSSE indicator
All
Freshmen
Seniors
B
SE
z
B
SE
z
B
SE
z
Higher order
1.04
0.55
1.88
1.14
0.66
1.72
0.27
0.8
0.33
Reflective
2.09∗∗∗
0.51
4.09
1.48∗
0.66
2.25
0.98
0.79
1.23
Learning strategies
−0.77
0.41
−1.84
−1.23∗
0.54
−2.27
−1.17
0.67
−1.75
Quantitative reasoning
−0.15
0.4
−0.37
−0.71
0.47
−1.51
0.82
0.62
1.31
Collaborative learning
−2.27∗∗∗
0.44
−5.22
−1.66∗∗
0.61
−2.72
−1.82∗∗
0.58
−3.14
Diverse others
1.19∗∗∗
0.32
3.74
0.66
0.43
1.54
1.26∗∗∗
0.39
3.21
Student faculty
−1.0∗
0.4
−2.49
−0.93
0.54
−1.71
−0.37
0.63
−0.59
Effective teaching
0.37
0.44
0.85
0.93
0.57
1.61
−0.96
0.67
−1.44
Quality interactions
0.22
0.47
0.46
−0.55
0.56
−0.98
0.26
0.76
0.34
Supportive environment
−0.21
0.4
−0.53
0.64
0.49
1.29
0.46
0.61
0.75
Constant
934.55∗∗∗
21.59
43.29
926.55∗∗∗
26.32
35.2
1043.17∗∗∗
37.03
28.17
Note:∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001; all estimated constants are significant at p<0.001.The next phase of analysis was driven by the results above, whereas only the items measuring those engagement indicators (EI) that were found to have coefficients different from zero would be included in the subsequent analysis. This lower level analysis examined the relationships of the individual items that compromised each of the four engagement indicators: Discussion with Diverse Others, Reflective and Integrated Learning, Student-Faculty Interactions, and Collaborative Learning. The descriptive statistics for the lower level items are reported in Table5. For a complete definition of the items listed in Table 5 see http://nsse.indiana.edu/html/summary_tables.cfm.Table 5
Descriptive statistics for engagement item level regression variables across all, freshmen, and senior class students.
Engagement indicator items
N
M
SD
All
Fresh
Senior
All
Fresh
Senior
All
Fresh
Senior
Collaborative learning
Ask for help
731
423
308
2.54
2.67
2.36
0.88
0.83
0.94
Explain to others
726
420
306
2.72
2.76
2.67
0.84
0.79
0.89
Study together
717
416
301
2.54
2.62
2.41
0.99
0.93
1.05
Group project
717
413
304
2.66
2.69
3.0
0.89
0.87
0.91
Reflective and integrative
Integrate
713
410
303
2.76
2.59
3.0
0.88
0.89
0.82
Societal
706
409
297
2.72
2.54
2.96
0.89
0.86
0.88
Diverse
706
405
301
2.7
2.58
2.88
0.88
0.85
0.89
Own view
711
409
302
2.88
2.79
3.0
0.79
0.79
0.78
Perspective
709
407
302
2.95
2.84
3.09
0.79
0.79
0.76
New view
705
407
298
2.95
2.86
3.07
0.77
0.77
0.77
Connect
704
405
299
3.07
2.92
3.28
0.76
0.76
0.72
Student faculty interaction
Career
706
406
300
2.61
2.68
2.52
0.93
0.89
0.97
Other work
702
404
298
2.08
2.17
1.95
0.99
0.95
1.03
Discuss
705
406
299
2.35
2.37
2.31
0.94
0.94
0.94
Perform
704
403
301
2.67
2.75
2.56
0.91
0.88
0.93
Diverse discussions with others
Race
680
392
288
3.14
2.98
3.35
0.91
0.91
0.88
Economic
674
387
287
3.05
2.89
3.25
0.91
0.9
0.89
Religion
676
389
287
2.91
2.77
3.09
0.98
0.96
0.98
Political
669
386
304
2.91
2.76
3.12
0.98
0.93
0.97
Note: highest score is 4.0.It was decided to continue to use maximum likelihood estimation here as with many survey-derived data sets; missing values occur randomly and therefore limit sample sizes when convention OLS regression is employed.The first EI to be examined at the individual item level was that of Collaborative Learning—or the frequency of how often student collaborated with others as measured by four different behaviors. As seen in Table6, the results indicate that not all behaviors contribute to the prediction of the CLA score. When considering both groups of students, three of the four items that comprise this EI had coefficients significantly different from zero. It appears that, as students ask others more frequently for help, CLA scores are predicted to decline. However, when looking across groups, studying with others fails to retain a significant coefficient. Asking for help and explaining to others retain significant estimators for both, with senior students having larger estimates. To determine whether the estimators were significantly different from each other, using equation (1) again, we found that z=0.84 and −0.91, revealing that the differences reported were not significant for either item across student class. It is suggested that these be interpreted as measures or indicators of student preparedness or understanding of the material. That is, as students need more help from peers, it indicates lower confidence or perceived ability, yielding the negative value for the predictors. Explaining to others conversely maybe an indicator of mastery of material or that explaining to others reinforces knowledge, skills, of confidence in the student’s perception of themselves as learners. In relation to each other, asking for help appears to exert the greatest influence on the CLA in terms of the items for this indicator for all students, and this relationship appears to change when groups are considered, albeit slightly. A one-unit change in explaining to others was predicted to increase CLA scores about 1.5% and 2.4% for freshmen and seniors, respectively, small but potentially influential.Table 6
Regression results for CLA total score and collaborative learning items with robust standard errors.
Collaborative learning
All
Freshmen
Seniors
B
SE
z
B
SE
z
B
SE
z
Ask for help
−41.31∗∗∗
7.18
−5.76
−24.51∗∗∗
8.37
−2.93
−35.36∗∗∗
10.27
−3.44
Explain to others
31.72∗∗∗
7.76
4.09
24.02∗∗
9.69
2.48
37.29∗∗∗
10.82
3.45
Study with others
−20.91∗∗∗
7.38
−2.83
−15.81
9.43
−1.67
−12.99
10.73
−1.21
Group projects/assignments
5.08
8.21
0.62
−2.09
10.76
−0.19
−10.25
11.51
−0.89
Constant
1036.93∗∗∗
19.87
52.19
977.48∗∗∗
11.51
39.18
1086.41∗∗∗
26.99
40.26
Note:∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001; all estimated constants are significant at p<0.001.The second engagement indicator analyzed was that of Reflective and Integrative Learning (RI) reported in Table7. When all students were included, two of the seven items were found to have significant predictors. The group analysis is quite revealing as not one of the items for freshmen has significant estimated coefficients, while for senior students, four items that range from moderate to highly significant predictors. This may reflect the value added from the educational experiences accumulated by the senior students. Conversely, freshmen had yet to benefit from the educational environment to the university and therefore may have been reporting based on prior experiences.Table 7
Regression results for CLA total score and reflective and integrative learning items with robust standard errors.
Reflective and integrative
All
Freshmen
Seniors
B
SE
z
B
SE
z
B
SE
z
Integrate
8.87
8.0
1.11
−4.45
9.48
−0.47
7.79
11.71
0.67
Societal
−3.19
9.19
−0.43
−0.33
12.85
−0.03
−22.99∗
10.58
−2.17
Diverse
9.83
9.35
1.05
−8.47
12.77
−0.66
25.78∗
13.18
1.96
Own view
−3.32
9.99
−0.33
7.32
11.83
0.62
−2.88
15.46
−0.19
Perspective
−3.18
9.96
−0.32
−7.74
11.15
−0.69
−8.04
17.61
−0.46
New view
−22.16∗
10.93
−2.03
8.43
12.95
0.65
−40.64∗∗
15.43
−2.63
Connect
45.94∗∗∗
11.34
4.05
15.69
11.46
1.37
58.68∗∗
19.82
2.96
Constant
881.82∗∗∗
22.71
38.83
897.37∗∗∗
27.8
32.28
979.25∗∗∗
36.06
27.16
Note:∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001; all estimated constants are significant at p<0.001.Senior class students were found to have the only significant results for any of the items defined as Reflective and Integrated Learning. The connect item asks students how often they connect ideas from their courses to prior knowledge and experiences. The reported positive relationship predicts that, for students who reported more opportunities to connect the learning to prior knowledge and experiences, their scores on the CLA were predicted to be higher. Therefore, as students believe they are more reflective and integrate prior learning with new material and courses, students score higher on the CLA and they engage more in such thinking. In fact, this was the strongest relationship found both in terms of frequency and percent of items. Also, the coefficient for connect with others was the largest found where a one-unit change was predicted to improve CLA scores by 3.8%.The next item with significant estimated coefficients is diverse perspectives, where an increase in discussions representing diverse views was found to predict higher CLA scores. Intuitively, this appears correct as being exposed to different views should cause you to be more reflective and consider more information before forming opinions or solving problems.The negative predicted coefficients for societal (connecting your learning to societal problems or issues) and new view (how a being exposed to a different view caused you to form a new one) are curious. It would seem that either of these would prompt different and deeper thinking in students; yet, this is not reflected in improved CLA scores. It just may be that the CLA offers problems and measures thinking that fails to utilize such skills.The next EI that analyzed in this manner was that of Discussion with Diverse Others (DD) with the results displayed in Table8. For all students of the four items for this indicator, only discussions with others as defined by race was found to have a coefficient different from zero. This may not be surprising for students attending this campus where the student population is fairly homogeneous across wealth, religion and political perspectives. For example, 63% of the student body is black, while economically 97% of students are classified as in need of financial assistance. While no data are available for political and religious diversity, all indicators appear to support the claim of homogeneity. Therefore, the most frequent opportunity for interaction with diverse others is by race, which as reported below may provide students opportunities to encounter opinions and beliefs defined across racial boundaries given that there are differences between how the two classes of students have this indicator, predicting CLA performance. The results are different when reported across the two student groups with freshmen having two items with significant predictors and seniors none. The signs for the two coefficients appear to convey very different perceptions of how economic and political diversity are translated into CLA scores. Discussion with others from different economic statuses was estimated to improve CLA scores about 2.3% for freshmen.Table 8
Regression results for CLA total score and discussion with diverse others items with robust standard errors.
Diverse others
All
Freshmen
Seniors
B
SE
z
B
SE
z
B
SE
z
Race
22.92∗
9.86
2.33
10.05
11.15
0.9
30.15
19.24
1.57
Economic
22.09
11.67
1.89
30.6∗
13.49
2.27
−6.61
21.65
−0.31
Religion
−7.44
9.69
−0.77
−7.61
11.09
−0.69
−6.96
15.77
−0.44
Political
−8.02
10.68
−0.75
−23.95∗
11.99
−2.0
7.49
18.93
0.4
Constant
884.69∗∗∗
17.61
50.25
900.28∗∗∗
22.22
40.5
962.34∗∗∗
25.52
37.72
Note:∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001; all estimated constants are significant at p<0.001.It is seen as freshmen more frequently reported discussion of educational experiences with those of different economic means, CLA scores are predicted to increase while the converse was reported when discussions with those with different political views are part of the students’ discussions. Freshmen may be reflecting in their scores previous life experiences, and given the nature of those experiences, how they are reflected in CLA scores may indicate the need to engage in such activities.The final group of items to be analyzed were those from the EI of Student-Faculty Interactions. The results seen in Table9 show that, for all students in the sample, the items addressing students working with faculty other than course work (committees, student groups, etc.) was found to have coefficients different from zero.Table 9
Regression results for CLA total score and student-faculty interaction items with robust standard errors.
Student and faculty
All
Freshmen
Seniors
B
SE
z
B
SE
z
B
SE
z
Career plans
8.56
7.54
1.13
8.11
8.96
0.9
14.21
10.52
1.35
Other work
−18.86∗∗
7.16
−2.63
−4.59
8.54
−.054
−19.14∗
9.53
−2.01
Discuss classwork
3.66
8.07
0.45
−9.39
8.85
−1.06
2.61
13.91
0.19
Discuss performance
−13.67
8.32
−1.64
−8.01
10.25
−0.78
−9.14
11.37
−0.8
Constant
1023.82∗∗∗
18.13
56.47
964.24∗∗∗
23.19
41.57
1062.73∗∗∗
23.76
44.73
Note:∗p<0.05; ∗∗p<0.01; ∗∗∗p<0.001; all estimated constants are significant at p<0.001.This was also true for senior students but not freshmen. The negative predicted coefficient may point to the fact that, as students engage in activities not directly associated academic classwork, CLA scores are estimated to decline. This decline may result from the additional time away from studies as a result of the commitment these activities require. Or that the nature of these activities is such that they do not provide experiences that improve the type of thinking and problem-solving as measured by the CLA. Or stated differently may reveal that as students were more involved in work outside the classroom, the demands these activities presented as far a time and effort, the return to critical thinking as measured on the CLA was negative for about a 1.2% decrease in CLA scores.There were nineteen items explored in this part of the analysis and on seven occasions when both groups of students were included significant predictors were estimated. For freshmen, this happened four times (21%) and seven (37%) for senior students. Only twice (5%) did both senior and freshmen have significant predictors (asking for help (−) and explaining to others (+)). Clearly, two patterns were revealed from both analyses (1) that for freshmen and seniors, the relationship between these NSSE items and CLA performance is different; (2) there was little agreement between the two groups of students; and (3) overall the relationship between NSSE and CLA is tenuous, complex, and may be of limited value.


## 5. Discussion

While the literature is saturated with studies reporting the linkage between engagement and beneficial student experiences, it seems plausible that associations should exist between the items surveyed on the NSSE and student performance measures. While such a linkage has been demonstrated on broad measures such as retention and graduation rates, linking the results from the NSSE to other measures of student learning remains elusive [11, 44]. This current study of the relationship between NSSE engagement indicators and CLA scores yielded outcomes that were consistent with those of the work of Gordon et al. [11] when they found that the NSSE indicators were not reliable predictors of student GPA, and this study extends this outcome to another measure of student performance.Given this, the results presented herein call into question the acceptance of the hypothesis presented by Carini et al. [46]. Their hypothesis stated students with higher ability, as measured by tests such as the SAT, may benefit less from student engagement elements. Gordon et al. mentioned this in their study as students in their sample had an average SAT score of 1340, implying that the relationship between NSSE measures and GPA would be weak as a result of the high mean SAT scores. However, students in this study had a mean score SAT of approximately 861. If students with lower ability are to benefit more from engagement, then the weak relationship between these two measures is curious. More completely, if the above were present, it is suggested that lower ability students would show stronger relationships to the NSSE indicators and that these indicators would be better predictors of GPA.The authors in the Carini et al.’s study proposed a term called “educational efficiency” that suggests institutions may experience a variation in how the engagement indicators associate with various learning outcomes. As a reference, the Gordon et al.’s study reported of the twelve opportunities for the NSSE benchmarks to correlate with GPA; it occurred 4 times, and in this present study, this did not occur for any of the ten indicators. Therefore, the proposed relationship between lower ability students benefiting more from engagement may not be present. In fact, it seems that, for this group of students with SAT scores some 450 points below of those in the Gordon et al.’s study, there are is a much weaker overall relationship between these measures. The measures were all consistent for all freshmen and senior class students.Results of previous research attempting to link the NSSE benchmarks and measures of student learning have shown weak associations as presented in the work of Carini et al. [46, 53, 55, 56]; therefore, it is not surprising in the current study that the reported relationships between the current NSSE engagement indicators and CLA performance was limited. Only four of the ten engagement indicators were seen as a reliable predictor of performance on the CLA. While this is better than the relationship between the NSSE and GPA, the association between engagement and achievement appears to be infrequent. Examining the item level findings reveal that, although there were four indicators found to be significant predictors of CLA performance, the connections are limited. The engagement indicator of Collaborative Learning having two of four items showing significance is consistent with what Fiorini, et al. found albeit again in their study that associations were found on the more global measures of student success such as graduation.Research on the effect of diversity experiences on student self-reported gains in various measures of cognition is well-documented [57, 58]. In their study, Pascarella extended this work to use a standardized measure of critical thinking and found results supporting the original work and revealing any such relationships were complex. Therefore, it was not surprising to find that the indicator Discussions with Diverse Others was a significant predictor of CLA performance. While not mirroring the complexity developed by Pascarella, the results presented herein show that this relationship is limited in scope as few of the item-specific measures yielded significant relationships with performance. Specifically, for all students, only discussions with diverse others defined by race proved to be a reliable predictor of CLA performance, and this was different for freshmen and senior students. Although two different approaches were used to investigate the idea of diversity and critical thinking, a degree of commonality was found. This lends support that students do benefit from interacting with others different from themselves. However, in this case, it was limited to freshmen, therefore implying that the college experience has a possible moderating effect on this interaction as a result of this university being more homogeneous than many high schools from which the freshmen graduated from.The point being the paucity of relationships found in this study between the NSSE and the CLA illustrates the difficulty and complexity of finding measures of student learning and student behaviors that can be measured in such a way to provide beneficial information to administrators, policy-makers, and other stakeholders. This is problematic as intuitions continue to rely on student self-reports as measures of behaviors, learning, and perspectives of the quality of education they receive exam-based [59].


### 5.1. Implications

The results are somewhat consistent with those of the larger studies that developed associations between ways to measure institutional effectiveness in terms of student learning as this study exclusively examined an HBCU with high percentages of Black students as compared to predominately white institutions (PWI). The results appear to offer that institutions serving minority students face similar issues as they strive to construct approaches to determine what institutional and faculty-based practices contribute to improvements in student learning. Taking this further, race or even gender may not be the student characteristics associated with attempting to understand how engagement should be considered as a tool to refine learning environments. Developing this, if students of diverse racial backgrounds experience the effects of engagement in a similar manner, then possibly, it is the school context that matters the most and not the individual student which aligns with the arguments posited by Kuh et al. [60]. It has been presented that [61] using data derived from student self-reported approaches may yield little in terms of information upon which to base improvement efforts. This may account for some of the perplexing results reported earlier. The potential for students to interpret the intent of a question fail to accurately remember personal and academic experiences; therefore, basing perceptions on faulty or incomplete information all can contribute to biased and inconsistent results and large numbers are needed to support efforts to comprehend how engagement is manifested in student outcomes.Aside from the issues associated with self-reported data, what policy implications does this study suggest? If institutions are interested in using data to drive improvements, a well-designed plan needs to be implemented that (a) either uses a probabilistic random sampling process to improve the potential to generalize or require key surveys or assessment activities to include all students, (b) invests in institutional research efforts focused on improving student outcomes, and (c) supports faculty involvement to engage in assessment activities as faculty are directly responsible for the types of engagement and learning activities that may have the potential in effecting student outcomes. This study did match the individual student survey and performance data; however, limitations in generalizability may be attributed to the sampling processes used by the university. While the analysis did show that including student demographic characteristics failed to improve the power of the models, this may have been more a result of sample size and not model misspecification as the researcher did face sample size limitations such as attempts made to develop alternative models.


### 5.2. Recommendations

Researchers and policy-makers are plagued with a multitude of technical and practical issues especially when examining the utility of attempting to associate student learning and institutional behaviors at the higher education level [38]. As Porter [59] recommends, assessing student learning with instruments such as the NSSE is problematic at best, arguably inappropriate, and should be discontinued. While possibly well-suited to link to gross measures of institutional performance such as retention and graduation rates [44], the NSSE benchmarks appear to be not well-suited in associating with specific measures of student learning; specifically, the CLA. The root cause of this can be argued, regardless the conclusion appears to be well-supported by this study and others.Where does this leave researchers and policy-makers interested in understanding those actions an institution can take to improve learning (quality) not just graduation (quantity), as many private and public sector members are questioning and demanding that institutions clearly demonstrate the worth of a college education [62, 63]. Therefore, searching for alternatives to assessing what defines a successful the college is complex and varied therefore utilizing such metrics as returns on investment [64] remain relevant. Taking this into consideration, measuring the quality of an education and the potential worth is critical for students and society. It is important to continue to research how institutions can not only demonstrate that they do produce quality graduates but also must understand how they did it.


## 5.1. Implications

The results are somewhat consistent with those of the larger studies that developed associations between ways to measure institutional effectiveness in terms of student learning as this study exclusively examined an HBCU with high percentages of Black students as compared to predominately white institutions (PWI). The results appear to offer that institutions serving minority students face similar issues as they strive to construct approaches to determine what institutional and faculty-based practices contribute to improvements in student learning. Taking this further, race or even gender may not be the student characteristics associated with attempting to understand how engagement should be considered as a tool to refine learning environments. Developing this, if students of diverse racial backgrounds experience the effects of engagement in a similar manner, then possibly, it is the school context that matters the most and not the individual student which aligns with the arguments posited by Kuh et al. [60]. It has been presented that [61] using data derived from student self-reported approaches may yield little in terms of information upon which to base improvement efforts. This may account for some of the perplexing results reported earlier. The potential for students to interpret the intent of a question fail to accurately remember personal and academic experiences; therefore, basing perceptions on faulty or incomplete information all can contribute to biased and inconsistent results and large numbers are needed to support efforts to comprehend how engagement is manifested in student outcomes.Aside from the issues associated with self-reported data, what policy implications does this study suggest? If institutions are interested in using data to drive improvements, a well-designed plan needs to be implemented that (a) either uses a probabilistic random sampling process to improve the potential to generalize or require key surveys or assessment activities to include all students, (b) invests in institutional research efforts focused on improving student outcomes, and (c) supports faculty involvement to engage in assessment activities as faculty are directly responsible for the types of engagement and learning activities that may have the potential in effecting student outcomes. This study did match the individual student survey and performance data; however, limitations in generalizability may be attributed to the sampling processes used by the university. While the analysis did show that including student demographic characteristics failed to improve the power of the models, this may have been more a result of sample size and not model misspecification as the researcher did face sample size limitations such as attempts made to develop alternative models.


## 5.2. Recommendations

Researchers and policy-makers are plagued with a multitude of technical and practical issues especially when examining the utility of attempting to associate student learning and institutional behaviors at the higher education level [38]. As Porter [59] recommends, assessing student learning with instruments such as the NSSE is problematic at best, arguably inappropriate, and should be discontinued. While possibly well-suited to link to gross measures of institutional performance such as retention and graduation rates [44], the NSSE benchmarks appear to be not well-suited in associating with specific measures of student learning; specifically, the CLA. The root cause of this can be argued, regardless the conclusion appears to be well-supported by this study and others.Where does this leave researchers and policy-makers interested in understanding those actions an institution can take to improve learning (quality) not just graduation (quantity), as many private and public sector members are questioning and demanding that institutions clearly demonstrate the worth of a college education [62, 63]. Therefore, searching for alternatives to assessing what defines a successful the college is complex and varied therefore utilizing such metrics as returns on investment [64] remain relevant. Taking this into consideration, measuring the quality of an education and the potential worth is critical for students and society. It is important to continue to research how institutions can not only demonstrate that they do produce quality graduates but also must understand how they did it.



---

*Source: 3648318-2019-04-22.xml*