# Presentation: 775736

## Using the Webinar as a Substitute for In-Class Security Training

**FISSEA 2004**
**March 11, 2004**
**E. Jane Powanda**

## Agenda

**What is a webinar**
**Producing a webinar- the ITSC experience**
**Using the webinar in a training program**
**Final thoughts**

## What is a Webinar

**Synchronous Internet training**
**Geographically dispersed participant learn together with out leaving home**
**Visual information via the workstation browser**
**Voice information via teleconference or voice over IP**

**An online presentation featuring an expert in a particular field using a web based collaborative tool.**
**(ASTD – American Society for Training and Development)**

## Developing and Delivering a Webinar -   The ITSC Experience   -

## ITSC

**INFORMATION TECHNOLOGY SUPPORT CENTER**

**STATE OF MARYLAND**
**ACS COMMERCIAL SOLUTIONS**

**MITRETEK SYSTEMS**
**UNIVERSITY OF MARYLAND**

**Sponsored by the** **US Department of Labor**

**Webinar Study**

***ITSC***

**www.itsc.state.md.us**

**Notes:**
The webinar feasibility project is a follow-on to the e-learning study that was performed last year.  The project last year concluded that the webinar could be a cost effective alternative method for delivering ITSC technical training to UI staff.

## What is the ITSC

## Applying IT to State Workforce Agencies

**Internet**
**Call Centers**
**One-Stop Offices**
**Client/Server**
**EDI**
**Voice Response Units**
**Geographic Information Systems**
**Decision Support**
**Imaging**
**Telecommunications**
**Process Workflow**
**Security**

## Webinar Feasibility Project

**Determine the feasibility of delivering ITSC training via webinar **
**Determine acceptance of training method by state participants **
**Gain experience in developing an effective webinar using the Mshow webinar delivery tool**
**Limited to a single webinar production**

***Scope***

**Notes:**
The goal of this project is to determine whether the webinar delivery method is feasible and acceptable to state UI staff for delivering ITSC technical training.  Though the research was not directed at determining whether the method is feasible for states delivering their own internal training, the information presented may be of value to the states in planning their own internal training programs.  
The method for determining feasibility was to prepare a webinar of relatively short duration and then offer it to a small group group of state UI professionals.  This activity was intended to help us to achieve the other two goals which were to determine the acceptance of the webinar  training method by state participants and to gain experience in developing an effective webinar using one of the commercial tools/services available.  
The scope of the project was limited to developing and delivering a single webinar.  The topic chosen for the webinar was selected based on discussions with state partners on this project and with ITSC UI call center technical and management staff.  

## Organization of the ITSC Study Briefing

**Basic requirements**
**State partner participation **
**Participant evaluation results**
**Planning the webinar**
**Developing the webinar**
**Delivering the webinar**
**Webinar Costs**
**Lessons learned**
**Recommendations**

**Notes:**
There are eight major topics addressed in this presentation as identified below.
The role of the state partner participants in helping to make this study meaningful
The evaluation results that were received from webinar participants
What was involved in planning the webinar
What was involved in developing the webinar
What was involved in delivering the webinar
The cost of developing and delivering the webinar
Discussion of the many lessons that we learned on this project
Recommendations for future use of the webinar method for ITSC technical training

## Basic Requirements for a Webinar

- 3

- 1

- 2

- Web
- Conferencing

- Teleconferencing

- Creative
- Instructor

- Important message
- well prepared

- 4

## Features of Webinar Delivery Tools

**Application sharing**
**Guided Internet tours**
**Display PowerPoint presentations**
**Audience polling **
**Chat feature for student/teacher interaction**
**Question feature**
**Presenters speak from multiple locations**
**Record session for playback at future time**
**Participant registration features**
**No special software required of participant**

## Webinar Delivery Tools – Some Examples

## A View of a classroom – (WebEx tool)

*A View of a classroom – (WebEx tool)*

## ITSC State Partner Participation

**Provided input on good and bad features of webinars  they attended in the past**
**Provided suggestions for webinar topics**
**Voiced preference for webinar length**
**Attended ITSC webinar and assessed effectiveness**

**Notes:**
The state partners played a key role in helping us to focus our efforts on the topics of importance to potential ITSC webinar recipients. The call for state partners to support the webinar feasibility study went out on January 16, 2003. State UI staff that attended two or more ITSC training workshops were targeted for the study as one of the primary goals of the project was to determine whether the webinar style of training could replace the in-person workshop training that the ITSC had traditionally been delivering for the past six years.  
A teleconference was conducted with the partners on February 20. Goals were to:
Determine how much familiarity the partner states have with webinars 
Solicit opinions (both positive and negative) about webinars team members had attended 
Determine what the participants currently think about the webinar as a potential way for the ITSC to deliver technical training to UI staff in the states.
The group talked about their experiences in viewing other webinars.  Their experience ranged from mediocre to bad.  Some of the most common concerns voiced were that webinar content was poor or it was not what they expected, the presenters were bad speakers or not well organized, there were delays between the screen and the spoken part, the visuals were boring, and most of the webinars were marketing presentations. On the up side, one person had attended a webinar that was well done. Characteristics of this webinar were good technical quality, the moderator kept the webinar on track, Q&A was handled well and the wrap up was good.
In general people did not think that webinars would be a good substitute for 3 day ITSC workshops.  One concern was that it was too long to be in front of the computer. Another was that you would be distracted by work things as you watched the webinar from at your desk. Others that watched a webinar projected in a conference room along with others did not like the experience.  Suggestions for ITSC webinar topics were also made.  These are discussed in the webinar planning section.

## Partner Comments on Webinars Attended in the Past was Helpful

**Characteristics of a good webinar**
**Good technical quality**
**Moderator kept webinar on track**
**Question & answer session handled well**
**Wrap up was good**
**Characteristics of a bad webinar**
**Poor content or not what expected**
**Bad speakers or speakers not well organized**
**Too much marketing**
**Delays between screen and spoken part**
**Boring graphics**

## Participants were Geographically Dispersed

## Participant Evaluation Results

**Would recommend it to their colleagues **
**Felt comfortable with the delivery method **
**Easy to connect to the webinar using instructions provided**
**Enough opportunity to participate in the class **
**Material was useful and well presented**
**The instructors were well organized**
**Pace was good**
**Length of presentation was good**
**Some concerns with voice quality**

**Notes:**
Rather than start with how the project progressed, let’s start with the results.  In general, the webinar received high approval ratings.  The evaluation form with tablulated results is included in Appendix A.  The participants were given 5 choices to each question.  Possible answers were: Strongly agree, Agree, Neutral, Disagree, Strongly disagree.  
The participants gave generally favorable answers (no Disagree or Strongly disagree responses) to the following questions:
I would you recommend this class to my colleagues.
I felt comfortable with the delivery method.
It was easy to connect to the webinar using the instructions provided
There was enough opportunity for me to participate in the class
The material presented was useful
The topic was well presented by the slides and dialogue
The instructors were well organized 
The pace of the class was appropriate
The length of the presentation was appropriate 
The results were unanimous on this last question.  All participants gave the answer “Agree.”
There were negatives on the voice quality question as we had some static during the presentation.  The static was on the line of one of our presenters.

## Participant Approval Ratings (would you attend another webinar? )

**Key:**
**Approval ratings from 8 to 40**
**8 – All strongly disagree**
**40 – All strongly agree**
**Eight persons provided input**

**Notes:**
One of the goals of the project was to gauge the acceptability of the webinar method of delivering training by those receiving the training,  The chart in this slide depicts the approval ratings for webinar participants choosing to attend another ITSC webinar.  The ratings were calculated by assigning a 5 to every Strongly Agree answer, a 4 to every Agree answer, a 3 for a Neutral answer, a 2 for a Disagree answer and a 1 for a Strongly disagree answer.  Eight persons completed the evaluation form.  If all eight had answered Strongly agree to a single question, the approval rating for that question would be 40.
Everyone either Agreed or Strongly agreed that they would attend another one hour webinar delivered by ITSC staff.  For the three hour webinar option, the opinion started to shift with fewer Strongly agrees and Agrees, several Neutral responses  and one Disagree. For a one day webinar, we received more Disagree and Strongly disagree responses than Agree and Strongly agree responses and for the Three Day option, the negative opinions became more pronounced.
Based on these results, we concluded that people would choose to attend webinar delivered training as long as each class does not exceed three hours.  

## I had planned only to note the presentation process but was pulled in by the presentation and am very excited about this technology! The presenters were well organized (maybe too much) and it appears that you had rehearsed most of this. 

*It allowed me access to a seminar/training environment without the travel and expense of that travel.  The sound was very good and it allowed plenty of opportunity to participate. *

*It was much better than just a conference call. It seems like a valuable tool now that tight budgets have reduced travel budgets for face-to-face meetings** *

*It comes across better if it sounds conversational, rather than obviously being read from text.** *

*Its quick, fast and I’m at my desk.*

**Comments from Participants**

**There was static in the audio**

**I am a visual learner and had some trouble staying focused on a disembodied voice. It did get better as we went on, but I could not have lasted more than 1 hr! **

**Notes:**
This slide includes some of the written comments that were received on the evaluation forms.  Participants seemed to like the idea that they could participate in useful training without having to travel or even to leave their desk.  On the other hand, one participant did not like listening without seeing a real person.  Another observation was that the webinar was staged rather than being spontaneous.  Another noted that it appear to be well organized and sensed that it had been rehearsed.  

## The ITSC Trial Webinar

**The Lobby**
**The Tools**
**Polling the Audience**
**Evaluation form**

**Notes:**
The next few slides will give you some idea of what the webinar participants saw and did.  The presentation was delivered using a PowerPoint presentation that included both text and graphics.  The presentation was enhanced by several voice examples of speech recognition in action.  
The other parts which were unique to the webinar experience are described in the next four slides.  They are the lobby, the student communication tools, polling the audience and the evaluation form.

## Welcome

**  ****Host: Jane Powanda**

**  ****Presenters:**

** ****Ron Mains**

** ****Tim Meier**

**. . .to the ITSC’s Inaugural Webinar on Speech Recognition in the State UI Agency Environment.**

**The Lobby**

**Notes:**
This is the first thing that participants saw when they logged in to the webinar.  It was part of the show’s lobby.  Three slides were included in the lobby, and very similar to pre-movie ads or notices, rotated in succession.  These slides continued to appear until we started the show.  The other two slides used were provided the agenda and advised participants to stand by as the show would be starting shortly.  During this lobby period, we were able to greet the participants as they joined the voice portion of the webinar.

## The show will be starting shortly

**The Lobby**

## Tools - Engaging the Attendees is Key!!

**Notes:**
One of the challenges of delivering a webinar is maintaining contact with the students.  Not seeing them and them not seeing you adds a new dimension to the teaching experience.  There needs to be some way to engage them in the class, and some way for the instructor to determine whether they are relating to the subject matter being presented.  A toolbar showing some of the instructor aids that we used in the course of our trial webinar is shown on this slide.  This toolbar was displayed on each participant’s screen.  
The red arrow was created by using one of the leader’s annotation tools that comes with Mshow.  The arrow can placed on the screen to draw attention to a particular place on the slide.  It can be moved and it can be erased.  Other figures like boxes and circles can also be used to dynamically highlight things displayed on the screen.  This tool helps the inatructor to focus attention on a particular point.
The raise hand tool allows the participants that tell the leader that they would like to speak.  It is up to the leader to determine whether on not to let the participant speak.  
The message tool permits the participant to send a message to the leader(s) the entire group or privately to another participant.  When setting up an Mshow the leader has the option of enabling or disabling private communication between the participants. A participant is notified of an message in their box by a flashing message icon.
The audience tool permits the leaders as well as the participants to view who else is attending the session.  A map view option presents this same information in a different format.  A map of the United States is displayed and the name of each participant currently logged in is placed on the map with an arrow pointing to the state in which they work. 

## Polling the Audience

**Are you currently using IVR for the first portion of your Initial Claims script?**

**Notes:**
The polling feature enables the leader to gain information from the participants for use in the presentation and at the same time gives the audience a chance to voice their opinions and to actively participate in the class.  Once the poll is completed, the Mshow tool tabulates the results and presents them in pie chart format showing percentage of the audience that selected each of the answers provided.  The polling question on this slide was presented to participants early in the webinar.  The “Results” shown is not indicative of the results received during the actual webinar. 
When creating a polling question, the instructor may choose a Yes/No answer format, or may use a multiple choice format.

## The Evaluation Form

**Notes:**
An evaluation form was developed to solicit information and comments from participants on their webinar experience.  The evaluation form which is displayed in its entirety in Appendix A was designed to gather information gauging the comfort level of participants with webinar presentations and to determine how open they would be to having training delivered to them using the webinar method.  In addition to answering multiple choice questions as shown on the slide, the respondents were also given the opportunity to “write in” comments to provide us with more information.
The evaluation form was placed on the ITSC web site and class participants’ browsers were automatically directed there when they exited the webinar.  (A feature of the Mshow tool enabled us to to this.)  Once the participant submitted the form (via the Internet), the ITSC team leader received the comments via e-mail.  All comments were received within 10 minutes of webinar completion.

## Planning the Webinar

**Work with state partners**
**Select a topic**
**Determine length of webinar**
**Organize the ITSC webinar team**
**Select webinar delivery tool**
**Research webinar techniques to determine how to produce a good webinar**

**Notes:**
The first step in planning the webinar was to determine how state UI staff in general felt about webinars, thereby giving us an idea of the some of the problems we may have to overcome. We established a team of volunteer state staff who were regular attendees at past ITSC workshops, and solicited their help in the following four areas.
Determine whether state UI staff would like to see this type of training, 
Determine the type of training that could be effectively delivered using the webinar,
Assess the effectiveness of webinar delivered training by participating in a short (1 hour) test webinar that will be broadcast to study participants only, 
Provide feedback on this webinar experience to the ITSC team..
Five states responded to the call as identified earlier.  Tennessee also became part of the team by participating in, and providing feedback on the webinar production. 
The next step was to organize the ITSC webinar team.  Because all state partner participants were call center experts, we decided that the topic of the webinar should have something to do with call centers or call center technology.  Two persons from the  ITSC call center group were added to the team to develop the content for the webinar. 
The next task was to determine which tool would be used to deliver the webinar.  Since Mitretek, our parent company, already had a contract with Intercall, we decided to use Mshow, the webinar delivery product provided by them.  Our staff conducted research to determine what makes a good webinar.  Webinar based training is a relatively new area, there is little printed documentation available.  Out best resource was watching other webinars.  In addition to watching webinars to identify good and bad delivery styles, we also viewed three webinars on how to produce a good webinar.  These were hosted by Interwise, Placeware and WebEx.  All provided some good guidance on both planning the webinar and delivering the webinar.

## Developing the Webinar

**Notes:**
Based on the interests of our state partner volunteers, the topic chosen for the webinar was speech recognition.  Relying on technical research on this topic already performed by ITSC staff, the PowerPoint presentation was prepared much like it would have been done for a traditional ITSC workshop.  A script was also prepared to go along with the presentation. We felt that the script was necessary so that the webinar would run smoothly and would be completed in the one hour time slot that we allocated for the webinar.  Based on our viewing other webinars, we made the decision that the webinar should be presented by several people rather than just one.  This adds variety to the program and helps to give the participants the benefits of several experts rather than just one. More information on the challenges in preparing  the content is presented on the next slide
The next step was to learn to use the webinar tools to develop the webinar.  There were some learning aids provided by Mshow, but these did not answer all of our questions.  Experimenting with the Mshow delivery tool and consulting with Mshow technical experts helped us to come up to speed on effectively using the product.  More information on this experience is provided in a subsequent slide.
The webinar was scheduled for August 20, and invitations were sent out to prospective participants.  The Mshow tool generated a draft invitation based on information we had input to it to formally schedule the webinar.  This draft invitation included the title, time (indicating correct time for multiple time zones), how to connect to the webinar and a short description of the webinar. We added additional information to this draft and e-mailed it to invitees.  Two follow up reminder e-mails were also sent: one was sent one day before the webinar; the other was sent two hours before the start time.
Reviewing and rehearsing the webinar took a significant amount of time (about 5 dry run rehearsals).  
Resolving technical issues was an unexpected task that required significant time in researching and resolving the problems. Some of these technical issues are discussed under Lessons Learned.
In preparing the evaluation form, the goal was to determine whether participants would take an ITSC class that was delivered via webinar.  Questions that solicited their feelings about the topic, the organization, presentation, and general satisfaction with the webinar were prepared.  We also provided a method for them to be able to provided unstructured feedback (written comments).

## Preparing the Content

**Topic chosen based on interests and needs of intended audience **
**Goals for content**
**Relevant, **
**Interesting **
**Timely and useful**
**Goals for presentation**
**45 minutes in length**
**15 minutes for introduction and Q & A session**
**Multiple presenters**
**Design slides**
**Write script for presentation**

**Notes:**
New speech recognition technologies was chosen as the topic because we felt that it would be of interest to our state partner representative audience, who were all involved with call center operations. 
In preparing the content, we set several goals for ourselves:
The content had to be interesting, relevant and useful to recipients
It had to be written such that it could be delivered by two or more different persons
It had to be no longer than 45 minutes in length
The content had two components, a PowerPoint slide show and a script.  The decision to use a script was made for mainly two reasons: we wanted to make sure that the presentation would be exactly 45 minutes (to leave 15 minutes for the introduction and a question and answer session), and we wanted to minimize the chances of appearing unorganized.  
For slide preparation, we based our style on what we liked about other webinar presentations that we had seen.  We found that the text on the slides needs to be fairly large with not too much information on the slide.  The text had to be legible, and the graphics used had to look professional.  One reason for not including too much information on a slide is that the viewers need to see new things appearing on their screen on a frequent basis.  Putting less on each slide and changing slides more frequently is good practice.
The script preparation was also a challenge.  We discovered that the script had to written as though it were a speech rather than a written narration of the slides.  The original script had to be modified to put it in this “speech” format.  It also had to be well coordinated with the text on the slide.  Without this text point of reference, webinar participants tend to get lost in the speech and can’t tell where you are in the presentation.  The slide change instructions and animation instructions were also included on the script.  This aid was necessary to help the person controlling the slides to ensure that slide changes and mouse click controlled animations were performed at the appropriate time. 

## Learn to use the tools

**Web training provided by Mshow**
**Help from Mshow tech support**
**Downloading plug-ins for conducting polls**
**Creating polling slides**
**How to tell when polling is complete**
**Navigating slide presentation**
**Controlling animation**
**Using the annotation tools**
**Self experimentation with the tool**
**At least five sessions with Mshow technical support persons to resolve technical issues**

**Notes:**
To learn to use the Mshow tools we downloaded the materials from the Mshow site and watched a recorded webinar providing more information on how to use the features and tools.  These features include audience polling tools, slide navigation and animation control, participant controls, and slide briefing annotation tools.  
To be able to insert audience polling in the presentation, we had to download a “plug-in” to be used with PowerPoint.  This plug-in enabled us to create question slides that would recognized and processed by the Mshow server.  One of the issues with the plug-in is that it introduces macros to your version of PowerPoint so “macros” must be enabled when you create an Mshow presentation.  We experienced some difficulty in creating polling slides as we modified their format to create more aesthetically pleasing slides than were created by the Mshow plug-in.  Any time that we modified a polling question or answer, we had to go back and re-do any formatting changes that we had made to the slide earlier.  One of the shortcomings of the Mshow tool is that when using the polling feature during a presentation, there are no indicators available to the presenter to let her know when the 30 second polling period is over.  This means that the presenter has to be aware of the starting time of the poll and must monitor the elapsed time clock.
There is a moderatly steep learning curve in becoming proficient with the Mshow annotation tools, and being able to use them effectively in a live presentation. These annotation tools include an arrow indicator, boxes to encase text and several other marking features.   We had moderate difficulty in learning to use these tools well.  Consequently for this first presentation, we refrained from making extensive use of the annotation tools.  We found that when we switched between presenters, the screen annotations could not be removed from the screen by the new presenter.  
The bottom line is that we spent a significant amount of time experimenting with the tools before we felt comfortable enough to use them.

## Delivering the Webinar

**Leaders on-line at least one half hour before webinar**
**Greet attendees as they join the webinar**
**Start on time**
**Welcome and give instructions for technical assistance during the webinar**
**Instruct attendees on use of tools and plan for the rest of the webinar**
**Pass control between leaders**
**Conduct Q&A**
**Log off and collect evaluation forms**

**Notes:**
The webinar delivery went as planned.  We had a final dry run on the day of the webinar and uncovered and corrected several last minute technical problems that we had not anticipated.. 
Based on earlier dry runs, we found that it was important for the leaders to be logged on about a half hour before the start of the webinar to verify that all are able to connect, to ensure that the voice volume for all speakers is consistent, and to be prepared to start on time.  Participants started to log-on about fifteen minutes before the scheduled webinar.  We greeted participants as they joined the teleconference and made it a point to start on time. 
The host welcomed the participants and gave instructions on how to access the tools and how to get technical help with webinar problems should the need arise.  There was one person who had trouble bringing up the web connection.  An Mshow technical representative was called upon to help the person.  There was also a problem with static on the line and we  upon teleconference technical help to identify the origin of the static so that the problem could be resolved.
Each of the three leaders presented their parts of the webinar from their own office and each was using a headset rather than a standard telephone. We had tried producing the webinar from a conference room but found that the voice quality was not good when using a speaker phone.  We also found that it was better for the presenter to be controlling their own slides rather than another person doing it for them.
As it came time for each presenter to speak, each took control of the web conference so that they could control their slides
The formal presentation took approximately 45 minutes, leaving 15 minutes for questions.  Questions were submitted to the host via the Mshow messaging feature.  The host moderated the Q&A session by reading the questions and enlisting the technical experts to address the questions.  Five questions were submitted and addressed.  The webinar met the time constraint of one hour and lasted 58 minutes.  As participants logged out, they were directed to an ITSC web site that contained the webinar evaluation form.  Participants completed the form and responses were immediately sent to the ITSC webmaster and the webinar project leader.

## Webinar Production Costs

- Webinar development
**One time cost for webinar development**
**PowerPoint slide presentation development**
**Script development**
**Rehearsals with total cast **
- Delivery cost
**Cost are incurred each time the webinar is delivered**
**These costs are per individual**
**Costs are incurred for both trainers and students**

**Notes:**
There are two components to webinar production costs: the development or fixed costs, and the delivery or variable costs.  
The webinar development cost includes the one time cost for the webinar.  The cost is the same whether the webinar is delivered once or is delivered 20 times.  It includes the effort for preparing the PowerPoint slide presentation, developing the script, and for rehearsing and refining the production.
The delivery cost included the staff time for the trainers and the cost of broadcasting the webinar.  This includes teleconferencing costs and web conferencing costs. These costs are incurred each time the webinar is presented.  The teleconferencing and web conferencing costs are incurred on a per person basis.

## Estimated Costs – Webinar Development

**One time cost for each webinar hour developed  $15,000**
**PowerPoint slide presentation development (40 hours)**
**Script development (40 hours)**
**Other (40 hours)**
**Rehearsal**
**Review**
**Problem resolution**
**Evaluation form**
**Training for persons developing or leading a webinar ( 20 hours) $2500**
**Total fixed cost $17,500 per webinar hour**

**Notes:**
The ITSC estimates that it costs approximately $15,000 in development costs for each one hour of webinar developed.  This includes 120 hours of staff time costed at $125 per hour.  For states are developing their own webinars using in-house staff, their labor costs would most probably be significantly less. 
The 120 hours includes 40 hours for preparation of a PowerPoint presentation that includes a mix of graphics and text, 40 hours to develop the one hour script that provides the narration for the slides, and 40 hours for review, rehearsals with all of the presenters, technical problem resolution and preparation and review of the evaluation form(s).
Note:  Fourty hours per classroom hour is typically our estimate for preparing materials for ITSC delivered classroom training. 
In addition to the preparation, presenters new to webinar class delivery require  some training on how to effectively deliver a webinar and how to use the webinar tools.  Approximately 10 hours of orientation per webinar speaker is required to get them oriented to webinar delivery methods. Assuming two leaders per webinar, this translates to 20 hours or $2500.  This would be a one time cost per webinar leader.

## Webinar Delivery Costs

**Staff time for trainers**
**Two staff hours per trainer for one hour webinar delivery**
**Assuming 3 trainers, cost is $750 per webinar hour**
**Cost for Web conferencing and teleconferencing services**
**$.38/minute per person connected – Mshow connection**
**$.21/minute per person connected – teleconference**
**Approximately $36/hour per person for webinar connection**

**Note:  The figures given for webinar delivery costs are based on “retail” cost.  It is possible to negotiate better rates for webinar services or even to pay a flat rate for unlimited monthly service.  This would significantly reduce the delivery cost per student hour.  **

**Notes:**
Three persons were used to deliver our webinar, a host and two technical persons delivering the featured presentation.  Each person had to spend about 2 hours on the day of delivery.  In addition to the one hour for actual delivery, an hour of preparation time is also required.  This translates to six hours of staff time.  At $125 per hour, total cost is $750.
The real driving force is the cost of the webinar connection along with the teleconferencing element.  Teleconferencing costs were $.21 per person per minute.  Web conferencing costs were $.38 per person per minute. This comes to nearly $36 per hour  per person.  This may be cost effective for short meetings but becomes very expensive for training that exceeds six hours in length.  At this rate, the webinar connection cost for an typical 18 hour ITSC workshop would be about $648 per person.
Compared to a weeklong trip than can cost upwards of $1000, the $648 for webinar connection costs could be considered cost effective.  If the training is only one day (8 hours), the cost would be $216 per day per person.  Assuming that someone would have to fly in for the training and spend 8 hours or more in travel time, the $216 for webinar costs would seem to be very cost effective. 
Lower webinar rates can be found by shopping around and negotiating with web conferencing service providers.  Some providers offer flat monthly rates that permit unlimited use of the webinar service.  If webinar delivery were to become the normal mode of operation and multiple webinars were delivered each month, a flat rate service could be very cost effective.  We have seen flat monthly webinar rates range from $40 to $3000 depending on the provider and services provided.  The rate for webinar hour could easily be reduce to $10 per webinar delivery hour per person.

## Cost Comparison: eighteen hour workshop – class size - 50

|  | Classroom | Webinar |
| --- | --- | --- |
| Development | $90,000 | $180,000 |
| Trainer (2) | $3750 * 2 = $7500 | $3000 * 2 = $6000 |
| Class materials | $750 |  |
| Travel cost | $50,000 | 0 |
| Room and refreshments | $3,840 | 0 |
| Connection cost 18/hr per student or trainer | 0 | $18,252 |
| Total | $154,090 | $204,452 |

**Notes:**
Looking at the costs a little differently, compare the cost of delivering an 18 hour ITSC workshop using the traditional method versus the cost of delivering it using the webinar method. When the training time is increased from 1 hour to 18 hours, the same people will be delivering all of the training so most of the rehearsal and learning curve time for the webinar option can be eliminated.  Here we assume $10,000 per hour for preparation of material to be delivered via webinar instead of the $15,000 per webinar hour that was discussed earlier.  We also assume that we can get web delivery services and teleconferencing for $.30/minute instead of the $.60/minute that the ITSC had to pay for a single one hour webinar.  With these new assumptions, cost calculations are explained below.
The development cost for  traditional classroom delivery is 18*40*$125 = $90,000 where each of the 18 hours of training takes 40 hours to develop and the rate for the developer is $125/hour. For the webinar development, 80 hours of development time is required yielding 18*80*$125= $180,000.
For classroom delivered training, trainers put in 10 hours a day (2 hours a day for travel).  For webinar delivery, the trainers would put in 8 hours per day. The rate for a trainer is $125 per hour
Class materials and the handouts are typically distributed at classroom training.  We estimate these materials to cost approximately $15 per person or $750 for the entire class. For the webinar training, the students would download materials themselves over the Internet.
Travel cost for traditional classroom training delivered in Bethesda, MD is estimated at an average of $1000 per student.
We estimate meeting room cost at $500/day and $15/day per student and trainer for refreshments. The meeting room and refreshments would be provided by a hotel or a for profit meeting facility.
Web conferencing cost and web conferencing costs are estimated at $.30 per minute of $18 per hour for each of the 50 students and two trainers 18*$18*52.

## Cost Comparison – 2nd class delivery of same material.  Class size = 50

|  | Classroom | Webinar |
| --- | --- | --- |
| Development | 0 | 0 |
| Trainer (2) | $3750* 2 = $7500 | $3000 * 2 = $6000 |
| Class materials | $750 |  |
| Travel cost | $50,000 | 0 |
| Room and refreshments | $3,840 | 0 |
| Connection cost 18/hr per student or trainer | 0 | $18,252 |
| Total | $62,590 | $24,252 |

**Economies of scale can be realized for the webinar option when it is presented multiple times, or the number of students is high**

**Notes:**
When the same class is delivered for the second time, there is no cost for development of class content.  All other costs remain the same.  A cost saving of about $38,000 is realized for the webinar delivery method. If the class were given a third time, an additional $38,000 cost saving would be realized.  When  the class is delivered three times, the webinar delivery method is more cost effective than the classroom delivery method. For two or less class deliveries, the classroom delivery method is more cost effective.
The same cost savings would be realized if the class sizes were increased, as the travel cost, room and refreshment cost and class materials cost increase proportionately with the number of students.  The major recurring cost for the webinar option is the connection cost and this is about 60% less than the cost for classroom training travel, refreshment and materials.  

## Cost comparison:  3 hour seminar – class size is 50

|  | Classroom | Webinar |
| --- | --- | --- |
| Development | $15,000 | $30,000 |
| Trainer (2) | $625*2=$1,250 | $500 * 2 = $1,000 |
| Class materials | $125 |  |
| Travel cost | $30,000 | 0 |
| Room and refreshments | $820 | 0 |
| Connection cost 3/hr per student or trainer | 0 | $2,808 |
| Total | $46,995 | $33,808 |

**Notes:**
For shorter ITSC classes or seminars that require that the participants travel, the webinar delivery method is more cost effective than the classroom delivery method.  A cost saving of approximately $13,000 would be realized by delivering a three hour seminar for 50 participants using the webinar method.  The method for reaching the costs is described below.
For a three hour seminar, the content development would take 120 hours.  At $125/hour, the cost would be $15,000.  Development of the same content for the webinar is estimated at 240 hours or $30,000.
For the classroom delivery method, the trainers would spend 5 hours each, since travel to and from the facility is required for each of them.  For the webinar delivery method, the trainers would spend 3.5 hours each since they need to be on line at least one half hour before the webinar.
Class handouts are estimated at $125.  For the webinar, the participants would download materials over the Internet.
Training room cost is estimated at $300 and refreshments are estimated at $10 for each student and trainer.  
Connection costs for the webinar are estimated at $.30/minute per student and trainer or $18/hour.

## Cost Comparison – 2nd class delivery of same material.  Class size = 50

|  | Classroom | Webinar |
| --- | --- | --- |
| Development | 0 | 0 |
| Trainer (2) | $625*2=$1,250 | $500 * 2 = $1,000 |
| Class materials | $125 |  |
| Travel cost | $30,000 | 0 |
| Room and refreshments | $820 | 0 |
| Connection cost 3/hr per student or trainer | 0 | $2,808 |
| Total | $32,195 | $3,808 |

***Cost Comparison – 2******nd****** class delivery of same material.  Class size = 50***

**When travel is required for classes of short duration, the webinar delivery method provides significant cost savings**

**Notes:**
When the three hour seminar is delivered to 50 students a second time, significant cost savings are realized using the webinar delivery method.  Over $28,000 in savings is realized.  The travel cost is the driving factor in this example. 

## Webinar Cost Conclusions

**The webinar is cost effective for short meetings where out of town travel is required**
**The webinar is cost effective for long training sessions (18 hours over a three day period) where out of town travel is involved, when either of the following conditions is met**
**The class is delivered multiple times**
**The class is delivered to a large number of students**

**Irregardless of cost, study participants indicated that they would not attend an 18 hour ITSC workshop webinar conducted over a three day period.  They may attend a 3 hour session, and would attend a 1 hour session.**

**Notes:**
The webinar delivery method can be cost effective for conducting ITSC training or meetings when travel is involved for the participants.  For local training (where no student travel is involved), the cost of the web and telephone conferencing would play a major role in determining whether the method is cost effective.  If local meeting rooms are available at no cost to an organization, there is little advantage to using the webinar method for training delivery.
The webinar delivery method is more cost effective for short training sessions or meetings than it is for long ones because the travel cost for a short meeting is proportionately a larger part of the total per student cost. 
The webinar method can be cost effective for long training sessions when the class is delivered multiple times (at least two) or the number of participants is high (over 150).  This is because of the high development costs for webinar delivered training.  This high development cost can be offset somewhat if web and telephone conferencing costs can be reduced.  If webinar training becomes the norm rather than the exception, flat rate monthly plans can be negotiated to obtain web and teleconferencing services at around $1,000-$3,000 monthly.  These flat rate plans generally require signing an annual contract for service.  This could significantly reduce the per student connection cost.
Special note: Irregardless of cost, feedback from study participants indicated that they would not be interested in attending an 18 hour ITSC workshop webinar conducted over a three day period.  They may attend a 3 hour session, and most definitely would attend a 1 hour session.  

## Lessons Learned

**Content is key to getting good results – present timely information **
**Scripting makes it easy for anyone to deliver a webinar**
**Spontaneity can be lost by adhering strictly to a script**
**Multiple speakers helps to add variety to the presentation**
**Speakers must work as a team**

**Webinar development and delivery takes more time than expected**

**Notes:**
The webinar experience presented us with many surprises.  The bottom line is that we found that it takes more time to produce a webinar than we expected.  Part of it was the learning curve in getting presentation information ready for webinar delivery.  The other was learning to use the Mshow tools.
The next few slides will describe some of the lessons we learned in the course of preparing a delivering a webinar.
Content is Key to getting good results – present timely information.  We had already learned from state partners that the thing they did not like about webinars that they had previously experienced was that the information was not useful to them.  Consequently we strived to find a topic of interest and to present it at the appropriate technical level.  The topic presented  provided participants with very current information not readily available to them. 
Scripting makes it easy for anyone to deliver a webinar. When the text for the presentation is prepared, a last minute substitute speaker can easily deliver the presentation. And scripting also can make a mediocre and unorganized speaker look good.
Spontaneity can be lost when the presentation is being read rather than being “taught”.  One of the things that students like about “live” teaching is the spontaneity.  With the use of polling and on the spot annotating the slides, some element of spontaneity taken away by reading a script can be put back into a presentation
Multiple speakers help to add variety to the presentation.  Based on our viewing of other webinars, we preferred the ones that had several speakers rather than just one.  The fact that you cannot see the speaker is most probably one reason this preference is common.
Speakers must work as a team.  Moving seamlessly between speakers makes the difference between a mediocre delivery and a well planned one. Using vocal handoffs helps to ensure that the next speaker comes in on time.  There are also cases where the technology fails and one of the leaders may need the support of the other leaders. We experienced software crashes during live presentation, so it it important for another presenter to be prepared to control the slides for a colleague.

## More Lessons Learned

**Audience participation is essential**
**Change visuals more frequently than for an in-person presentation **
**Ensure that slides are readable**
**The visual must be matched with the spoken part**
**Problems with the technology can be expected**
**Have a backup computer in case one fails – one will!**

**Notes:**
Audience participation is essential..  It is easy to lose students that you cannot see.  We found that using the Mshow audience tools provided help mitigate this problem.  The polling questions and the question and answer sessions were the two things that participants liked best.
Change visuals more frequently than for an in-person presentation. Because the speaker cannot be seen, it is necessary to frequently change the screen image to keep the participant visually stimulated. Some people learn by hearing, others by seeing.  For those visual learners, it is important to use change the screen frequently, even it it means only moving an arrow to point the the information currently being discussed. Use of animation also helps to achieve this goal.
Ensure that the slides are readable.  Because of individual system configurations and PC setup, the size of the slide that each participant views may be different.  Consequently small print may not be readable by some participants. The slides need to be designed for the “worst” case.
The visuals must be matched with the spoken part.  During rehearsals we found that when the script did not precisely match the text on the slide, the participants tended to get lost.  We modified either the script or the slide to resolve this.  Another alternative is to use the arrow annotation tools to point to the part of the presentation that is currently being discussed.
Problems with technology can be expected. The technology for delivering webinars is not perfect.  Early on we saw what could happen when a vendor demonstrating his system to us had to abort the presentation because his computer crashed.  We experienced a similar situation when a presenter’s browser software, used to control the webinar, crashed.  This same type of crash also occurred on a participant’s machine at a different time. 
Have a backup computer in case one fails – one will.  One of the advantages of having all presenters in different rooms is that each can have access to a computer and control the webinar from that computer.  We had one instance where the browser of one presenter crashed during the presentation, taking away his ability to advance to the next slide. Another presenter was able to take control, advance the slide, and our presenter continued talking as if no problem occurred.

## More Lessons Learned

**Speaker phones don’t work for presenters! **
**Use individual telephone headsets for presenters**
**Speakers don’t need to be in the same room**
**Adjust headsets and volume to ensure consistent voice quality among presenters**
**Give instructions on how to get technical help at beginning**

**Notes:**
Use individual telephone headsets for presenters – speaker phones don’t work.  During a dry run where the three presenters were staging a webinar from a conference room using a speaker phone, the feedback from our reviewers was that the voice quality was poor (“you sound like you’re talking in a fishbowl”) Using the headsets has several advantages – the voice quality is generally better because background noise is reduced, and it also blocks out distracting noises for the speaker who is trying to concentrate on delivering his presentation.
Speakers don’t need to be co-located.  Once we started to use the headsets, we realized that there was no need for the speakers to be in the same place.  This opens the possibility for including a speaker that is in another state.
Adjust headsets and volume to ensure consistent voice quality.  During rehearsals we found the different speakers came across louder and clearer than others.  It it critical that speakers log on at least 30 minutes before Showtime to allow time for adjusting the headset positioning and the speaker volume.
Give instructions on how to get technical help at the beginning.  We learned this one the hard way.  The instructions for getting help  were at the end of our introduction rather than the beginning consequently there was some disruption during the introduction with getting problems logistic connection problems resolved.

## Still More Lessons Learned

**Use a production manager**
**Promotes consistent presentation development**
**Helps participants resolve technical problems during the webinar**
**Collects questions for Q & A session**
**Providing online capability for completing the evaluation form increases the likelihood of getting feedback**
** ****Webinar delivery costs can be expensive**
**Negotiate good rates for Webcast service **
**Use identical equipment for rehearsals and live show**
**TEST TEST TEST**

**Notes:**
Use a production manager.  We found that it was helpful for one person to coordinate the production both throughout the development phase and during the live presentation.  The style of the presentation needs to be tailored specifically for webinar delivery and the production manager worked with content developers to get the presentation in proper format.  During the presentation, the production manager collects questions and deals with any problems that students are having with the presentation such as connection problems, voice quality, etc.
Providing on-line capability for completing the evaluation form online increased the chances that you will get timely feedback.  The Mshow tool gives you the option of directing the participant’s browser to a specified web site when then exit the presentation. We placed the evaluation form on a ITSC web site and had participants directed to that web page.  The result was that we received 6 evaluation forms within 10 minutes after the show completed.
Webinar delivery costs can be expensive.  Retail rates for webinar delivery are high.  We paid $.21 per minute for teleconferencing services and $.38 per minute for webcast delivery services.  This adds up to nearly $.60 per minute per person or about $36 per hour.
Negotiate rates for webinar service.  Doing comparison shopping on webinar service providers can help to bring delivery costs down.  Look for flat rate monthly plan if you plan to do at least three or four webinars a month.  Otherwise, try to negotiate acceptable per minute rates.
Use identical equipment for rehearsals and the real thing.  We found that that new technical problems tend to surface when we moved from the test configuration to the production configuration.  For testing, we typically used our local phone service rather than a teleconferencing service provider.  Voice quality was different using the teleconferencing service and we had to make adjustments to deal with it.  Rehearsal using the telephone headsets was also important as was having the presenters control their own presentation rather than depending on a co-worker to change slides for them..
TEST TEST TEST.  We had heard this advice from others and can’t reiterate enough the importance of it.  If something can go wrong, it will.  Extensive testing will surface most of the common problems that can occur.  Having experienced them once, you will want to put together contingency plans to deal with all of the things that could possibly go wrong during a live delivery.  

## Using the Webinar in a Security Training Program

## Recommendations for Using the Webinar as a Replacement for Classroom Training

**Schedule workshops over a two to four week period rather than over several days**
**Offer sessions independently rather than packaged as a complete workshop**
**Develop short seminars on timely security topics **
**Record the webinar for future playback**
**Limit class size to encourage student participation **
**Use guest speakers from within or  outside the organization**

## Some Things Show Better Using a Webinar

***Some Things Show Better Using a Webinar***

**Walk through a hack scenario**
**Screen shots difficult to see when projected in a classroom**
**Demonstrate the use of a program or tool**
**Backup procedures**
**Vulnerability assessment tool**
**Security Self Assessment Tool**
**Show real vulnerabilities by surfing the web and viewing HTML code**
**Screen shot of code difficult to read when projected in a classroom**

## Webinar Training - Implications for Presenters 

**Better preparation necessary – students less forgiving of a teacher they cannot see**
**No long pauses in speech**
**Short modules are best**
**Follow up activities enhance learning**
**Need to poll students for feedback**
**Students need materials ahead of time**

## Modifying Classroom Training Materials

| Time | Topic |
| --- | --- |
| 9:00 – 10:15
Session 1 | Threats, Vulnerabilities & Risk
“It happened Here” – Panel |
| Break |  |
| 10:30 – 12:00
Session 2 | Common Security Mistakes – Video
Introduction to Security Solutions
State Policy/ Incident Handling
People and Procedures  |
| Lunch |  |
| 1:00 – 2:30
Session 3 | Technology Solutions
State Network Security Implementation
Scams, Fraud, and Spam |
| Break |  |
| 2:45 – 4:00
Session 4 | About Forensics
Physical and Personal Security |

***Security Awareness Training ***
***Re-organize into six 45 minute webinars***

## Modifying Classroom Training Materials

| Time | Topic |
| --- | --- |
| 9:00 – 10:15
Session 1 | Introduction
Security in the Application Lifecycle
Security Basics
Writing Security Requirements |
| Break |  |
| 10:30 – 12:00
Session 2 | Writing the Operational Security Plan
Web Programming Issues  |
| Lunch |  |
| 1:00 – 2:30
Session 3 | Authentication Options
Assessing Security Risk
Closing Remarks and Questions |

***Modifying Classroom Training Materials***

***Security Analyst Training ***
***Re-organize into five one hour webinars***

## Use Free or Inexpensive Security Webinars 

**Offered by vendors, not for profit associations and others**
**Free or pay per view**
**Live or Recorded**
**Educational or marketing**
**Some Examples**
**Intrusion Detection Essentials - SANS Institute**
**Managing Security Incidents in the Enterprise  - Symantec**
**Dynamic Threat Protection  -  ISS**
**Securing .NET Applications from the Start – Microsoft**
**Identity Management Strategies for HIPAA Compliance  -  RSA Security**

## Examples of Free Webinars

## Examples of Free Webinars

***Examples of Free Webinars***

## Final Thoughts

**The Webinar is a cost effective training alternative when:**
**Students are geographically dispersed**
**The webinar is well done**
**The webinar is short enough to hold student interest **
**The webinar is recorded and used multiple times**
**Webinars may not work in secure environments**

## Contact Information:

- Jane Powanda	_[jpowanda@mitretek.org](mailto:jpowanda@mitretek.org)_
- 				_[jpowanda@itsc.org](mailto:jpowanda@itsc.org)_
- 				301 513-0143