Improving Student Learning

An Evaluation of theEffectiveness ofGraphically Enhanced
Learning Materials
Executive Summary

This study was conducted by SEG Research, an independent educational research firm located in New Hope, Pennsylvania. SEG
Research provides research, evaluation, and assessment services to educational publishers, educational technology providers,assessment service providers and government agencies. SEG has been meeting the research and assessment needs of organizations
since 1979. This research was supported by a grant from John Wiley & Sons, Inc.

EXECUTIVE SUMMARY
Background and Purpose
During the fall semester of 2009, SEG Research conducted a national,
multi-site study of students enrolled in entry-level college Psychology,Geology and Environmental Science courses.1 The study was designed toevaluate the effectiveness of Wiley Visualizing. Wiley Visualizing includes
both graphically-enhanced textbooks and digital media, and is designed
to better engage students and improve student learning and retention
through to course completion. Wiley Visualizing makes extensive use ofvisual pedagogy to enhance the student learning experience.

Students who use Wiley Visualizing perform betterin their course than do students who do not use aVisualizing book. Students using Wiley Visualizing
did 20-25% better on the posttest than thosestudents using a competing textbook.
The experiment involved over 1,200 students
and 17 instructors at 12 different institutions.
The students were enrolled in either a Psychology,or Geology course.

The goal of this study was to evaluate theeffectiveness of Wiley Visualizing. The study
compared the growth in content knowledge
and skills among students using Wiley
Visualizing and a comparable group ofstudents who did not use Wiley Visualizing.
The findings indicate that students using
Wiley Visualizing made significantly greater
gains in content area knowledge and skills
over the course of a semester than students
in classes that did not use Wiley Visualizing.
Study Design

DEFINITION:
Treatment Group = classes that used Wiley
Visualizing Control Group = classes that did
NOT use Wiley Visualizing.
Students in the Control Group used competing
texts including:•Jordan / Grotzinger, The Essential Earth•

Tarbuck, Earth Science•

Withgott, Environmental Science•

Hockenbury, Psychology•King Essentials, Psychology

During the fall semester of 2009, 1,203 students
enrolled in courses taught by 17 instructors at
12 different institutions from across the UnitedStates participated in a controlled study of theeffectiveness of Wiley Visualizing. Classes thatused Wiley Visualizing during the course of thestudy constituted the Treatment Group andthose classes that did not use Wiley Visualizing
constituted the Control Group. There were
779 students in the Treatment Group and 287
students in the Control Group. Students inthe Treatment and the Control Groups weresimilar in ability and background.

Environmental Science was eliminated from the study due to small sample size and attrition.

1

Using a quasi-experimental2, pre-post design, this study compared thegrowth in student content knowledge and skills of students in classes using
Wiley Visualizing (Treatment Group) and a comparable group of students
who did not use Wiley Visualizing (Control Group). Student growth incontent knowledge and skills was measured by comparing scores from a50-item multiple choice measure constructed based on a survey of topics
commonly taught in the introductory course administered at the beginningand the end of the semester.
Students in both the Treatment Group and the Control Group took thepretest at the beginning of the semester (August/September 2009) toobtain a baseline measure of student content knowledge and skills.
Students in the Treatment Group used Wiley Visualizing during the courseof the semester, while those in the Control Group used Textbooks otherthan Wiley Visualizing. At the end of the semester (November/December2009), students in both the Treatment Group and the Control Group wereadministered the posttest. The results from the pretest and posttest wereanalyzed to determine the level of growth in content knowledge and skills.
The study controlled for any initial differences in the initial knowledge andskill levels of students in the Treatment and Control Groups using analysisof covariance (ANCOVA).

DEFINITION:
Quasi-Experimental = A study comparing two or more groups where initial differences in the groups areadjusted statistically.
Pre-post design = A statistical method where students are tested both before and after a event (such astaking a course).
“Growth in content knowledge and skills”: The students were tested to measure the effect of Wiley
Visualizing on tests measuring their knowledge and skills. Taking a class improves the student test scoresby 20-25%, compared to students using competing textbooks. By testing the students before the class,
we can ensure that there is no difference at the start of the study in the ability among students in thetreatment and control groups.

For a copy of the design, please contact SEG Research at: SElliot@SEGMeasurement.com

2Summary of Results
Pre-Post Growth for students using Wiley Visualizing
Students who were in classes that used Wiley Visualizing showed substantial
growth from pretest to posttest in both Psychology and Geology. During thecourse of the study, Psychology students in classes using Wiley Visualizing
increased their scores by nearly 6 points (5.97) or about 12%. (Meanpretest=16.40; Mean posttest score=22.37; see Figure 1). The posttest
scores were significantly higher (t=16.55; df=1,277; p<.001). The effect size
was .98.
During the course of the study, Geology students in classes using Wiley
Visualizing increased their scores by about 5 points (5.27) or about 11%.
(Mean pretest=17.13; Mean posttest score=22.40; see Figure 1). The
posttest scores were significantly higher (t=8.479; df=1,97; p<.001). The
effect size was .86. This indicates that students using Wiley Visualizing
showed substantial growth in content knowledge and skills during thecourse of the semester.
While the growth achieved by students using Wiley Visualizing is an
important indicator of effectiveness, a more complete way to assess growthis to compare the growth achieved by students in classes using Wiley
Visualizing to students in classes that did not use Wiley Visualizing. This
allows us to see the unique contribution using Wiley Visualizing made tostudents’ growth.

Students who use Wiley Visualizing perform better in their course than do students who do not use
Wiley Visualizing. Students using Wiley Visualizing did 20-25% better on the posttest than those students
using a competing textbook.
DEFINITION:
Effect size = A common metric for comparing the size of the difference in scores between two or moregroups that is attributed to the variable that is tested (in this case, using Wiley Visualizing).

Overall Comparison of Treatment and Control Group Growth
The overall growth in Psychology knowledge and skills by students in classes
using Wiley Visualizing (Treatment Group) was compared to the growth incontent knowledge and skills of those students in classes not using Wiley
Visualizing (Control Group). Analysis of Covariance (ANCOVA) was used toevaluate the difference in the posttest results (dependent variable) between
Wiley Visualizing Users and those not using Wiley Visualizing (independentvariable) controlling for the initial pretest results (covariate). The pretest
scores were used as the covariate to place students in the Wiley Group andControl Group on the same baseline.

DEFINITION:ANCOVA = A statistical method that compares two or more groups, controlling for the different abilitiesof the students in the different classes.
Independent Variable = The set of variables that can be controlled in the experiment (e.g., the use ofWiley Visualizing).
Dependent Variable = The variable that we use to measure to see if there is a treatment effect (e.g., testscores).

There were significant differences in content knowledge and skill growth
between the Treatment and Control groups for both Psychology andGeology. Students using Wiley Visualizing showed much greater increases
in knowledge and skills through the course, than their counterparts that did
not use Wiley Visualizing.
The results show a significant difference in Psychology content knowledge
and skills posttest scores between the Treatment Group and the Control
Group (df=2,411; F=39.00; p<.001) when initial content knowledge andskills (pretest) are controlled (See Figure 2). The effect size was .58.
The results show a significant difference in Geology content knowledge
and skills posttest scores between the Treatment Group and the Control
Group (df=2,157; F=28.17; p<.001) when initial content knowledge andskills (pretest) are controlled (See Figure 2). The effect size was .73.

Conclusion
Students who used Wiley Visualizing achieved signiﬁcantly greater gains
in knowledge and skills than students who did not use Wiley Visualizing.
Substantial differences were found for Psychology and Geology.
Wiley Visualizing users showed significant gains in content knowledge
and skills from pretest to posttest (Psychology=12%; Geology=11%). More
importantly, students using Wiley Visualizing did significantly better thantheir peers who did not use Wiley Visualizing. Psychology students scoredabout 4 points (8%) higher than their peers on the content knowledge andskills assessment, an effect size of .58. Geology students scored more than
4 points (9%) higher than their peers on the content knowledge and skills
assessment, an effect size of .73.
These effect sizes (.58, .73) are substantial, indicating that the students whoused Wiley Visualizing performed better than those students who did notuse Wiley Visualizing. One way to interpret the magnitude of the effectsize is to consider the potential percentile gains that could be achieved byusing Wiley Visualizing. For a student at the 50th percentile, use of Wiley
Visualizing could increase the student’s score in Psychology to the 72ndpercentile or in Geology to the 77th percentile.
These ﬁndings are particularly signiﬁcant for two reasons: First, the study
was conducted for only a single semester, representing fewer than 15 weeksof instruction. Second, there are many influences on student achievement;that textbook choice accounted for such a large effect on student growth incontent knowledge and skill in this context, is particularly noteworthy.

Students who used Wiley Visualizing achieved significantly greater gains in knowledge and skills.
The impact of using Wiley Visualizing is significant in how students retain information in Geology andPsychology. Gains are seen in both absolute terms (i.e., the improvement from pretest to posttest) andrelative terms (i.e., the advantage of using Wiley Visualizing versus not using it).

About SEG
This study was conducted by SEG Research, an independent educational
research firm located in New Hope, Pennsylvania. SEG‘s mission is tomake a measurable difference in education by providing high-quality
research, evaluation and assessment services. SEG Research provides
services in K-12, higher education, and post baccalaureate educationand to higher education institutions, government agencies, educational
publishers, and educational technology providers.
SEG is a recognized leader in research and assessment and has been
meeting the needs of organizations since 1979. SEG has conductednumerous efficacy studies in areas ranging from English Language Learners
to Science for both text- and, technology-based solutions designed to help
students at all levels learn.

About Dr. Scott Elliot,
Principal Investigator
Dr. Scott Elliot served as the Principal Investigator for this research study.
He has served as President of SEG Research for the past five years andpreviously served in senior roles for several research and assessment
organizations. Dr. Elliot has directed numerous higher education assessmentand evaluation programs including teacher evaluation and assessment
programs used in 40 states and the Texas Academic Skills Program (TASP).
Dr. Elliot has been conducting educational research and developing
educational assessments for more than 30 years and is a recognized expert.
Dr. Elliot received his Masters Degree in instructional communication fromWest Virginia University and he received his Doctorate in educational policy,
research, and assessment from the University of Massachusetts, Amherst.
Dr. Elliot is a frequent presenter at educational conferences and has morethan 75 scholarly publications and presentations to his credit.

John Wiley & Sons, Inc.
and SEG Research