1





          1

          2

          3

          4

          5

          6

          7

          8          ***************************************

          9           U.S. ELECTION ASSISTANCE COMMISSION

         10                   ROUNDTABLE DISCUSSION

         11                     DECEMBER 11, 2007

         12          ***************************************

         13

         14

         15             UNITED STATES ELECTION ASSISTANCE

         16        COMMISSION ROUNDTABLE DISCUSSION was taken

         17        before April C. Balcombe and Michelle

         18        Robertson, Certified Shorthand Reporters in and

         19        for the State of Texas, reported by

         20        computerized stenotype machine at the Omni

         21        Austin Hotel Downtown, 700 San Jacinto

         22        Boulevard, Austin, Texas 78701, on December 11,

         23        2007 commencing at the hour of 1:00 p.m.

         24

         25



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  2





          1                    P R O C E E D I N G S

          2             >> KING:  Well, good afternoon everybody.

          3        We will get started here in just a moment.  I

          4        think all the panelists are here.  We have a

          5        few housekeeping items, if you would, the

          6        normal cell phones, Blackberries, anything that

          7        beeps, squawks, if you would either put it on

          8        silent or off would be appreciated.

          9             The agenda for this afternoon is a panel

         10        discussion on five issues related to security

         11        in the draft version of the 2007 VVSG, and we

         12        have brought together a panel today of people

         13        with both experience and interest in security

         14        issues related to voting systems.  And Matt, if

         15        you would, if you would flip to the next slide.

         16        Back up one.  As I was preparing for my

         17        comments today, I was doing my normal Googling,

         18        looking for precedent.  Always interested in

         19        historical perspectives on voting systems and

         20        technology in general, and I came across a

         21        couple that I thought might help at least me

         22        get my perspective on this.  And I think in

         23        fairness to Charles Dual, that is attributed to

         24        him, but I don't think he ever really said

         25        that.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  3





          1             In any event, Matt, if you would go to the

          2        agenda, this is an ambitious agenda.  It is

          3        what we are going to attempt to hold to today.

          4        We have got five questions that we have been

          5        asked to address, and we are going try get a

          6        break into the middle.  If we get fatigued we

          7        may break sooner, and we may break more often.

          8             But what I would like to start with is

          9        asking the members of the panel to introduce

         10        themselves, and then we will have opening

         11        remarks from Brian Hancock from the E.A.C.

         12             So Ron, if we can start with you and just

         13        work around the table.

         14             >> I am Ron Rivest.  I am on the computer

         15        science department at MIT.

         16             >> I am Peter Ryan.  I am a professor off

         17        computer science at New Castle University in

         18        the UK.

         19             >> My name Daniel Castro.  I am a senior

         20        analyst with the Information Technology and

         21        Innovation Foundation.

         22             >> I am Alec Yasinsac with SAIT Lab and

         23        computer science department at Florida State

         24        University.

         25             >> Okay.  Chris Thomas, Director of



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  4





          1        Elections, state of Michigan.  And Chair of the

          2        U.S. Election Assistance Commission Board of

          3        Advisors.

          4             >> Sarah Johnson, Executive Director of

          5        Kentucky Board of Elections and the Chair of

          6        U.S. Election Systems Standards Board Executive

          7        Board.

          8             >> Brian Hancock, I am Director of Testing

          9        and Certification for the U.S. Election

         10        Assistance Commission.

         11             >> I am John Wack from the National

         12        Institute of Standards and Technology.

         13             >> Good afternoon.  My name is Costis

         14        Toregas.  I have two hats here.  One is a

         15        computer science scientist in George Washington

         16        University, and I am also on the staff of

         17        Montgomery County, Maryland, advising the

         18        county council on information technology

         19        matters.

         20             >> Michael Shamos.  I am a professor in

         21        the School of Computer Science at Carnegie

         22        Mellon University, and I think I am here

         23        because I have done over 120 voting system

         24        exams for six states since 1980.

         25             >> Hello, I am Juan Gilbert from Auburn



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  5





          1        University, professor in computer science and

          2        software engineering.

          3             >> I am Merle King.  I am Executive

          4        Director of the Center for Election Systems at

          5        Kennesaw State University, and I will be

          6        moderating this panel discussion today.

          7             If you look at the agenda, you will notice

          8        we have got five questions spread out over five

          9        hours and if you have figuring the overhead, we

         10        will spend 30 to 40 minutes per question.  I am

         11        hoping that everybody will kind of

         12        self-regulate themselves, and we will share the

         13        microphone appropriately.

         14             When you do get ready to speak, if you

         15        would, either put your tent card up vertically

         16        or in some way get my attention, and we will

         17        try to move around the room as quickly as we

         18        can.  But before we begin, I would like to ask

         19        Brian Hancock to give some introductory

         20        remarks.

         21             >> HANCOCK:  Thank you, Merle, appreciate

         22        it.  And thank all of you for agreeing to

         23        participate with us here in this very important

         24        meeting.  We know you have busy schedules, and

         25        we know that many of you have come from quite a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  6





          1        long distance in order to join us.

          2             I should note that this is a first in a

          3        series of round table discussions that the

          4        E.A.C. intends to hold on the TDJC

          5        recommendations.  We will have similar sessions

          6        with test labs, accessibility and usability

          7        professionals, election officials, activists

          8        and manufacturers.  Roundtable discussions will

          9        each continue to educate and inform the E.A.C.

         10        on specific areas of the document under

         11        consideration.

         12             I also need to acknowledge Matt Masterson

         13        over there of the E.A.C. who did the majority

         14        of the work to make this roundtable discussion

         15        a reality.  Matt is also the primary E.A.C.

         16        staff person available and working on a

         17        day-to-day basis dealing with next iteration of

         18        the VVSG.  Before I turn the round table over

         19        to the capable hands of our moderator Merle

         20        King, I would say a few words about why we are

         21        here holding this discussion.

         22             Beyond the obvious, that is to get input

         23        on the TGDC recommendations for the next

         24        iteration of the VVSG, I think we need to look

         25        at what we are trying to accomplish in a macro



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  7





          1        sense.  General goals of course are to develop

          2        an implement standards to make voting systems

          3        as reliable, secure, accurate, and accessible

          4        as possible.

          5             These rather simple words when spoken are

          6        rather simple words when spoken but as all of

          7        you know better than I do, extremely difficult

          8        to produce.  By undertaking the development and

          9        adoption of a new iteration of the VVSG, the

         10        E.A.C. and its partners are charting the course

         11        for the development of voting systems in the

         12        United States for the foreseeable future.  This

         13        is a truly awesome responsibility and one which

         14        we must undertake with full commitment and

         15        proper due diligence.

         16             Given the scrutiny that this process

         17        specifically and the electoral process more

         18        generally is under, failure in this endeavor is

         19        certainly not an option.  This is why we have

         20        invited you here today.  The NIST and the TGDC

         21        work on these recommendations while extensive

         22        and very good is only the beginning of the

         23        process of development, review, and adoption.

         24        It is not the end.  The real work for the

         25        E.A.C. election officials, academics, and the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  8





          1        American public begins now.

          2             The questions we pose to the panelists as

          3        conversation starters related to fundamental

          4        aspects of the TGDC recommendations.  The

          5        questions were asked so we can begin open and

          6        reason discussion on the direction we're

          7        charting for our voting future.

          8             To open our round table, we ask the

          9        question related to the development of a

         10        detailed risk assessment framework for voting

         11        systems.

         12             I personally feel that the development of

         13        risk assessment should be the cornerstone of

         14        framing our public debate about the VVSG.

         15        Others in the academic community agree with

         16        this assessment.  Professor Douglas Jones of

         17        the University of Iowa has stated that, I

         18        quote, if we can quantify the costs of threats

         19        and defensive measures we'll be able to rank

         20        threats in order of their likelihood and

         21        defensive measures in order of their

         22        importance.

         23             Professor Jones acknowledges this will be

         24        a difficult task but an essential task not only

         25        so we can judge the adequacy of our voting



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                  9





          1        system standards but also the adequacy of our

          2        recommendations for best practices and the

          3        adequacy of state laws and administrative

          4        procedures.

          5             I thank you, once again, for joining us

          6        today and look forward to our discussions this

          7        afternoon.

          8             >> KING:  Thank you, Brian.  If you can,

          9        Matt, move to the first question.  I hope

         10        members of the panel as well as the audience

         11        can see the question as it is displayed.

         12             But reflecting back on Brian's comments

         13        about the need for a risk assessment, I thought

         14        when I first looked at voting systems, that

         15        analogous comparisons were really

         16        inappropriate.

         17             I would hear people say that voting

         18        systems should be like ATM machines or voting

         19        systems should be like lottery systems, and not

         20        only did I think that an analysis of those

         21        environments would not aluminate voting

         22        systems, I thought that may be harm done in the

         23        comparisons, because voting systems are

         24        somewhat unique.

         25             My opinion of that changed somewhat, in



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 10





          1        part, based on presentation of a gentleman from

          2        the Nevada Gaming Commissions IT Security

          3        Group, and it opened my eyes that maybe some

          4        other people are working with models that we

          5        can learn from and would be appropriate.

          6             So, the question about risk assessment --

          7        and I am going to throw some items out for our

          8        panel to kind of help form a starting point for

          9        this discussion -- is some questions.  And one

         10        is:  Have -- is there a consensus of the

         11        definition of a scope of a voting system?

         12             And for those of us who do systems

         13        analysis, scoping the system is always

         14        critical -- what is inside, what is outside of

         15        the system.

         16             Are there users?  Are there managers?  Who

         17        are the stakeholders involved in the systems?

         18        Have we defined the risk, and more importantly,

         19        have we kept up with metrics by which to

         20        measure risk?

         21             Risk is the possibility of an event

         22        occurring that has a negative impact on an

         23        organization, so we think about risk in terms

         24        of the likelihood of occurrence and the impact

         25        of that risk.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 11





          1             Have we distinguished between inherent

          2        risks -- those are risks in a system for which

          3        there is no control -- and residual risk, risks

          4        that still remain once controlled are designed

          5        into a system?

          6             Do our risk models presume a capability to

          7        audit all risk out of a system?  And if you

          8        work with auditors or if you have done auditing

          9        work, we talk about the cost of auditing risk

         10        out of a system as you approach the 98, 99, 100

         11        percent levels of certainty, the costs of

         12        auditing the risk approaches infinity.

         13             And then another item that I really don't

         14        see talked about in risk assessment of voting

         15        systems, but we talk about it all the time in

         16        financial auditing, is the notion of reasonable

         17        assurance.  Do we have a metric?  Is that in

         18        our lexicon voting systems of auditing to the

         19        point of reasonable assurance?

         20             In auditing the credibility of the

         21        auditors in developing the risk assessment

         22        models are critical.  In financial auditing, we

         23        see the auditors have credentials, they have

         24        experience sets, they have training, they have

         25        a code of ethics.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 12





          1             And the IIA, the Institute of Internal

          2        Auditors, one of the requirements to be an

          3        auditor is:  Shall engage only in those

          4        services for which they have the necessary

          5        knowledge, skills, and experience.  So even

          6        talking about who should be developing the risk

          7        assessment models can be a problem.

          8             In looking at how do we assign weights to

          9        risk, how do we prioritize risks, in financial

         10        systems we look at are the risks associated

         11        with core functions within the organization?

         12        Are they associated with ancillary?

         13             Are risks greater for -- for continuity

         14        plans; that is, contingency plans that enable

         15        an election center to keep functioning; or are

         16        the risks greater for disaster recovery plans?

         17             The extent of the system, the change, are

         18        you looking at re-engineering the system or

         19        making trivial changes to the report generation

         20        functions of the system?

         21             So there are many models in the financial

         22        world that enable auditors to identify risk, to

         23        develop models of risk, and very importantly,

         24        to reach consensus of those models for the

         25        stakeholders.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 13





          1             In the corporate model of risk assessment,

          2        particularly since the Sarbane Oxley

          3        legislation, the identification of who is

          4        accountable for risk assessment in the model,

          5        and then, finally, I think there was a question

          6        about what are the allowable levels of risk.

          7             And I am a pilot, so I look at the NTSB

          8        reports frequently, and there have been six

          9        incidents at the Austin, Texas airport in the

         10        past couple of years.  And I got a ticket to

         11        fly out of there tomorrow, so I guess, by

         12        assumption, that is an allowable level of risk.

         13             But we often don't talk in elections, we

         14        have a tendency to swing very quickly to the

         15        absolutes, and we don't talk about what is

         16        reasonable assurance, what is allowable risk.

         17             So with what as my questions for the

         18        panelists to kind of open the discussion, I

         19        think Ron also had some remarks that he wanted

         20        to open with.

         21             >> RIVEST:  Thanks, Merle.  I prepared

         22        some remarks that address both risk assessment

         23        and software independence.  I think it is

         24        related.

         25             I guess I should state first I am on the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 14





          1        TGDC, but my remarks today are my own.  They

          2        don't reflect the TGDC in any formal sense of

          3        the E.A.C. or anybody else, so I am happy to

          4        answer questions based on my experience in the

          5        TGDC and so on.

          6             The questions were raised, what are the

          7        risks in voting systems, how to assess them,

          8        what are acceptable and unacceptable risks.

          9             And I think the best guide we have to that

         10        so far is the Brennan Center Report on the

         11        machinery of democracy.  They did a careful

         12        study, looked at hundreds of different risks to

         13        a voting system, and tried to evaluate their

         14        severity.

         15             They identified software is the most risky

         16        component in voting systems.  The voting system

         17        may contain malicious code or code that is

         18        erroneous.  It can change votes and cause an

         19        incorrect election outcome, perhaps

         20        undetectably.

         21             Of course people are the source of most

         22        problems, and their metric, I thought, was a

         23        very interesting one, which is how many people

         24        are required to cause, say, an incorrect

         25        election outcome or a significant change in the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 15





          1        election outcome to happen.

          2             So if you're looking at what are you

          3        risking election, you're risking that we get

          4        some of the incorrect election outcome, if we

          5        don't get that right, then our democracy isn't

          6        working properly.

          7             And another risk, an associated risk, is

          8        that the outcome is right but it is not

          9        believed.  You don't have sufficient evidence

         10        to convince the loser or the populous or the

         11        voters that it is the correct outcome.

         12             So the TGDC proposed this notion of

         13        software independence as a way of mitigating

         14        some of the risks associated with software.  I

         15        want to talk a bit about it.  It is perhaps the

         16        most significant aspect of the draft of the

         17        VVSG.

         18             It is there to mitigate the risk of

         19        software problems causing incorrect election

         20        outcomes.  So "software independence" means

         21        basically -- quoting here -- "that an

         22        undetected bug or error or malicious code in

         23        the software can cause an indetectable change

         24        in the election outcome."

         25             Maybe it is more helpful to look at it the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 16





          1        other way around.  Software independence

          2        means that you are placing your complete trust

          3        in the counting of the votes by the software or

          4        accepting any risk that there may be incorrect

          5        or even malicious software in the counting of

          6        the votes.  So what is wrong with software

          7        independence?  We heard lots of testimony in

          8        the TGDC that indicate software is correct and

          9        not susceptible to manipulation is beyond the

         10        state of the art.

         11             This is obvious to people using the

         12        computers today.  Gene Spafford wasn't able to

         13        make it here.  He had some nice words that he

         14        put in his written testimony saying it really

         15        isn't possible to tell whether software or

         16        other technology contains what he called hidden

         17        functionality.  So is it possible that this

         18        thing might do something sometimes that is not

         19        what you expect?  That is just not part of what

         20        we know how to do.

         21             And if you look at it from an actually how

         22        you would assess it, it is really hard.  A

         23        voting system may contain 10,000 pages of code

         24        and any one page changing an I to a J somewhere

         25        may cause the program to misbehave at some



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 17





          1        point which may not be revealed in testing but

          2        may happen in an election.  This is

          3        particularly hard when the relevant software

          4        may not even be available to software systems.

          5        May depend on operating systems or drivers for

          6        which the source code is not even available for

          7        review by the tester.  So it really is a tough

          8        problem.  Be nice if it weren't so, but it

          9        really is hard.

         10             So our voting systems certification will

         11        probably never mean that the software in the

         12        voting system is bug free and always gives the

         13        correct result.  Guaranteed -- we just are not

         14        able to give that level of assurance these

         15        days.  In addition, we have the problems of

         16        assuring that the software on the voting system

         17        is indeed the software that was tested in the

         18        first place.

         19             California, I thought it was interesting,

         20        they have been top to bottom reviewed, revealed

         21        how easy it is on the existing voting system

         22        for a virus to propagate from one machine to

         23        another, changing the software and every

         24        machine that the voting memory card went to.

         25             So to mitigate these software independence



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 18





          1        requires that the election outcome not be

          2        totally dependent on records produced by

          3        software.  The voter must have the ability to

          4        directly verify his or her choices on a paper

          5        ballot, and these paper ballots must be usable

          6        as a check on the electronic counts, such as

          7        with a post election statistical audit.  Any

          8        time you are trying mitigate risk, you fall

          9        into one of two major paradigms.  You are

         10        either trying to prevent the problem, which is

         11        what you doing when you are trying to prove the

         12        software correct or detect the problem and

         13        recover from it and the paper ballots and the

         14        post election audit are really in that

         15        paradigm.  Software independence fits in the

         16        detect and recover paradigm.

         17             So the major concern is undetectable

         18        changes in the election outcome.  You have got

         19        to have a -- you don't want to be hood winked

         20        and not even know about it.  I would like to

         21        view software dependence as very much like

         22        being a seat belt in a car.  Cars may have

         23        undetected faults in the braking system or

         24        elsewhere causing you to end up in a ditch, and

         25        undesired outcome.  Seatbelts prevent you from



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 19





          1        the undesired outcome, you know, going through

          2        the windshield or whatever.  Of course, you

          3        don't need to wear the seatbelts.  You don't

          4        need to do a post election statistical audit

          5        but you should.

          6             There is lots of issues with software

          7        independence, I mean, voters may not examine

          8        all their ballots.  You need to consider that

          9        and realize that even if significant fraction

         10        of voters do look at their voter verified

         11        records that you do it in confidence that

         12        attempts by software to cheat the voters can be

         13        detected.  And you realize there are

         14        advantageous when the voters actually read the

         15        ballots themselves in that regard.  It is also

         16        the case that a post election statistical audit

         17        itself may not look at all the ballots.  It is

         18        after all, just a statistical sample.  But it

         19        turns out to be, if you do the math,

         20        surprisingly cheap.  I mean, if you have a five

         21        percent margin of victory, 20 percent recount

         22        will actually get you up to over to 90 percent

         23        confidence that you will have found fraud

         24        sufficient enough to change the election

         25        outcome.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 20





          1             Of course, an audit is not required by the

          2        draft of VVSG.  It can't be since the draft

          3        VVSG is just about equipment testing and not

          4        about procedures.  But a well-designed voting

          5        system based on the principle of software

          6        independence can yield election outcomes having

          7        a high degree of confidence that they are

          8        correct, that they correspond to what the

          9        voters' choices are.  The risk of incorrect

         10        election outcome is mitigated by the use of

         11        these voter verified independent records and

         12        the post election audits.

         13             Back to the original question about

         14        acceptable and unacceptable levels of risk, I

         15        personally think it is unacceptable for a

         16        voting system to have the property of a

         17        undetected software bug or error or malicious

         18        code can undetectable change the election

         19        outcome.  That is something we don't need to

         20        accept and shouldn't.

         21             I also think it is unacceptable for any

         22        single person to change the electronic outcome,

         23        you know, even before the post election audit.

         24        And OVET can check for these kinds of things.

         25             There are reasons for doing things like



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 21





          1        the OVET that went into vulnerability testing

          2        that are outside of the integrity of the

          3        election.  There are issues of voter privacy

          4        for example.  Can someone figure out how you

          5        voted from the records created by the -- that

          6        you can't detect just by looking at paper

          7        ballots.  You really need to look at the system

          8        and internal of that to tell whether voter

          9        privacy is being well protected.

         10             So there is lots of more to say about

         11        different kinds of risk.  There is, you know,

         12        risk that we will miss out on new developments

         13        and technology and so on.  We will be talking

         14        about that later.  I think I have talked long

         15        enough.

         16             >> KING:  Thank you, Ron.  With that, then

         17        I would like to open up the discussion to other

         18        panels and Mike.

         19             >> SHAMOS:  I think I would find it useful

         20        if we did a little more question defining

         21        before we got heavily into the topic.  So how

         22        do you evaluate what is an allowable level of

         23        risk?  That sounds like that it begs for

         24        numerical answer.  That we are willing to

         25        accept one in thousand or one in million



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 22





          1        elections going bad.  And I am not sure that is

          2        what the question -- that is what the question

          3        really is.  So I would like to start with a

          4        discussion of what we mean by an allowable

          5        level of risk and how it would every be

          6        assessed.

          7             >> KING:  Okay.  Do you have thoughts on

          8        that?

          9             >> SHAMOS:  Of course.  I am not proposing

         10        an answer to the question.  I can tell you

         11        about some experiences that we have had though.

         12        I was on the project serve review committee in

         13        2004 for a system for internet voting for

         14        Americans overseas, and one of the exercises

         15        that we attempted to go through, which we

         16        ultimately discarded as foolish, was an attempt

         17        to place probabilities on various potential

         18        actions.  So what is the chance that an

         19        intruder will try to effect the outcome of this

         20        election?  I don't think it matters whether the

         21        chance is one percent or 100 percent.  You want

         22        to prevent them from doing it.

         23             If it is an issue that we are doing risk

         24        assessment so we can decide where to spend the

         25        money and spend a lot of money preventing the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 23





          1        risks that we think are very likely to occur

          2        and then not spend very much money on the risks

          3        that we don't think that are likely to occur,

          4        that will increase the probability of the ones

          5        that we don't think are likely because once

          6        people figure out what we haven't protected,

          7        they will turn their attention to that.

          8             And I think the whole exercise of risk

          9        modeling, the fundamental folly of it was

         10        brought home to me in the recent mortgage

         11        crisis.  There was a company called Campbell

         12        and Company, which was an 11 billion-dollar

         13        hedge fund that lost 12 percent of its value in

         14        one day because of the failure of the mortgage

         15        market.

         16             And when the CEO of Campbell was

         17        interviewed by the Wall Street Journal as to

         18        how this could possibly happen, his answer now

         19        often quoted, you can see many hits on the

         20        internet on this.  He said, "Our risk models

         21        failed."  And so you can develop the greatest

         22        model that you think you can develop, but if it

         23        is inaccurate for some reason, it is going to

         24        only give you false assurance that you have

         25        really guarded against something.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 24





          1             So instead of -- what I do favor is an

          2        exercise in risk assessment where you attempt

          3        to enumerate as many of the possible risks as

          4        you can, not necessarily signing probabilities

          5        to them.  But then devise remediations for them

          6        and attempt to place a value or cost on making

          7        those remediations and then see what the whole

          8        picture is of weighing the necessity of

          9        evaluating intentions which is what you are

         10        doing assessing when you evaluate the

         11        probability of something occurring.  So I

         12        didn't answer the question, but just to get

         13        things started.

         14             >> TOREGAS:  Two additional thoughts on

         15        the risk question.  Do you, Mr. Moderator,

         16        encourage us to think about the scope that we

         17        feel comfortable with.  And I think if I

         18        thought of the American voter, he or she is not

         19        so concerned about only the machines risk

         20        profile but the risk profile of the entire

         21        process of elections.  And I know that the VVSG

         22        is looking at a machine, but perhaps we might

         23        think about how we could daisy chain risk

         24        models all the way up and down from end to end

         25        of the entire voting process and the entire



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 25





          1        election process so we could give that voter

          2        the confidence not only in the machine but also

          3        in the entire process.

          4             The second concern that I have, and again

          5        you mentioned it, that sometimes in a risk

          6        analysis, we say 97 percent, we can fund

          7        98 percent.  We begin to run out of money,

          8        99 percent, we are getting pretty close to

          9        infinite and so on.

         10             And the question therefore begs an issue

         11        that we discussed in another forum about a week

         12        ago, and that is the beginnings of some kind of

         13        a cost benefit analysis which would help us

         14        place the risk evaluation in the context of

         15        components other than probabilities alone, and

         16        you have mentioned one, which might be expense,

         17        cost, how much are you willing to pay to

         18        guarantee an extra nine on your string of nines

         19        after the 99 gets done?  And so those are the

         20        two thoughts I wanted to propose that we spend

         21        sometime on, cost benefit analysis, and the

         22        notion that the scope of the risk assessment

         23        ultimately has to be an end-to-end process of

         24        the election process itself so therefore as we

         25        are looking into the VVSG at the machine



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 26





          1        itself, how can we make it so that the risk

          2        model we discussed and the risk model someone

          3        is sent off to create can in fact be related to

          4        additional risk models up and down that chain.

          5        Thank you.

          6             >> KING:  John.

          7             >> WACK:  A bunch of points.  Everybody

          8        said very interesting things.

          9               One thing that I want to point out is

         10        that while a risk assessment, I think, is very

         11        well justified, very important, it shouldn't be

         12        used too much to arrive at decisions as to --

         13        for example, you know, is software independence

         14        justified or not.

         15             In my experience in this process, a number

         16        of people have said -- have come across with

         17        reasoning of the form of various things such as

         18        software independence and whether we will have

         19        to continue to using paper records is going to

         20        be expensive, do the risks, do the threats,

         21        really justify this, prove it.

         22             I want to point out, in the answers to

         23        these sorts of questions, there are a number of

         24        things that we have to keep in mind.  One is

         25        that, as everybody knows, voting system is



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 27





          1        complex information technology equipment.

          2             And in fact, you can even argue in some

          3        ways, because it is used infrequently and when

          4        it is brought up, it has to run correctly,

          5        things of that sort.  It might even be more

          6        complex than when we generally think of -- when

          7        we think of desktop systems in large

          8        organizations and complex networks.

          9             There has already been a fair amount of

         10        risk assessment work in this particular area.

         11        Where I work in the government we have been in

         12        charge of writing guidance for other agencies

         13        on how they ought to manage their networks and

         14        manage their information technology.

         15             And pretty much what is in the

         16        recommendations right now starts pretty much

         17        with the guidance that we have issued thus far.

         18        We have taken advantage of risk assessments in

         19        that particular area and have pretty much said,

         20        they apply to information technology, we think

         21        very much they apply to voting systems as well.

         22        So I would not like to see a risk assessment

         23        kind of start at the very beginnings and not

         24        make use of that existing work.

         25             The other thing that I would like to say



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 28





          1        is that, in an assessment of risks, I think it

          2        is important to not look solely at what sorts

          3        of threats are out there and the likelihood

          4        that they'll be exploited or what sorts of

          5        vulnerabilities, but also look at the usability

          6        of things.

          7             One of the other things that has come

          8        across in doing risk assessments in helping to

          9        manage systems more securely is to look at

         10        their usability.  Are they easy to manage?  If

         11        they aren't easy to manage, if the controls are

         12        difficult to use, they won't be used.

         13             If paper records are produced that contain

         14        all of the information you need for very

         15        precise audits, but the paper is thin, it tears

         16        easily, it jams in inexpensive printers, it is

         17        less likely that audits will occur.

         18             So the overall usability of the systems,

         19        the ease of managing them, the ease of using

         20        some of the audit capabilities that are out

         21        there today also have to be factored in, in

         22        this overall assessment, I think, to basically,

         23        ultimately arrive at what I believe the goal

         24        is, which is:  Shall we go ahead with certain

         25        things that have been proposed, such as



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 29





          1        software independence.

          2             So maybe I am a little bit all over the

          3        place here in my comments, but I guess mainly

          4        what I want to say is, I think many things have

          5        to be looked at in this assessment overall, and

          6        I would like to just caution that we take

          7        advantage of what is out there already, in

          8        terms of managing information technology, and

          9        also look very carefully at how well the

         10        controls that are out there today are being

         11        implemented and whether they are sufficiently

         12        usable.

         13             Sometimes making things easier to use for

         14        people makes them far more secure than

         15        otherwise.  That's really all I had to say.

         16             >> KING:  Okay.  Alec?

         17             >> YASINSAC:  Yes, thanks.

         18             Dr. Rivest mentioned the Brennan Center

         19        study, and that is an excellent study.  In

         20        fact, there are a couple of excellent studies

         21        from there.  And there are several sources, as

         22        John mentioned, of threat models out there.

         23        The California top to bottom review did an

         24        excellent threat model, and there have been

         25        several done at Berkeley and other places.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 30





          1             But from what I've seen, there has not

          2        really been anyone to try to put a structure on

          3        these things.  The Brennan Center study is a

          4        perfect example.  It is a wonderful study, very

          5        comprehensive, but it is all rules of thumb,

          6        it's all expert opinion, and there is very

          7        little quantification laid down to it.

          8             For example, there is no structure laid

          9        over those threats that would allow somebody to

         10        reason about classes of threats beyond the

         11        specific instance that Dr. Rivest gave of the

         12        ability of one or two -- a small number of

         13        people to have an impact.  That was the

         14        exclusive focus of that particular study.

         15             So how much impact a single line of code

         16        would have on the ability to conduct a safe

         17        election, there is really no way to reason

         18        about that with the models we have.

         19             And so in order to even conduct an

         20        effective study -- and we have done it Florida

         21        state, now six of these studies -- and in order

         22        to conduct an effective study, you have to have

         23        a much more precise, much more systematic

         24        classification system to be able to detail what

         25        the threats are so that the threats can be



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 31





          1        analyzed in the context of the system that is

          2        being evaluated, and those tools just aren't

          3        there right now.

          4             >> KING:  Thank you.  I wanted to come

          5        back to something that Mike said about the

          6        mitigation of risks, and I am curious, from our

          7        two election officials that are on our panel --

          8        because obviously, any deficiencies in risk

          9        mitigation cascade down and accumulate at the

         10        implementation level.

         11             I am curious whether you have any thought

         12        on the role of risk mitigation at the

         13        implementation level, as opposed to higher up

         14        in the development cycle and the design.  I

         15        don't want to put you on the spot, but --

         16             >> JOHNSON:  I think it is extremely

         17        important to have it at that level, too.  I

         18        guess we're sort of -- like the individuals

         19        have already said, a cost benefit analysis is

         20        something that is very important to us, because

         21        we obviously have to, you know, afford to

         22        purchase them and keep them up, and also what

         23        John is saying the usability and trying to

         24        marry those two, keeping all of that in mind,

         25        when you're developing models, the assessments,



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 32





          1        and the classification system is very important

          2        to us, as election officials, because we have

          3        to take that system and obviously go back to

          4        our local, county individuals and then our, you

          5        know, precinct officers, who are predominately

          6        elderly, to be able to use these machines, much

          7        less the voter.

          8             We see the forest, and it is a pretty big,

          9        thick, dense forest right now.

         10             >> THOMAS:  I would concur with Sarah.

         11        What we have up there are methods of avoiding

         12        risk that come from older systems, and we're

         13        now bringing new systems in, so there is this

         14        process of trying to adapt the older checks and

         15        balancing and the other assessments.

         16             The higher up that is done, I think that

         17        is a good approach, but when it ultimately

         18        comes down to the implementation, which is with

         19        all of these folks -- but that's the next level

         20        that is really going to integrate these new

         21        systems into what our current laws, procedures,

         22        and whatnot have to avoid the risks.

         23             >> KING:  Alec, I wonder if you have a

         24        comment on that, because I think Chris has just

         25        added another dimension to your observation;



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 33





          1        and that is, the legacy threat models may no

          2        longer be appropriate to the new technologies

          3        that we're rolling at.

          4             >> YASINSAC:  Well, I think that is

          5        absolutely true.  It really boils down to the

          6        notion that led to software independence, is

          7        that changing -- electronic changes can be

          8        very, very easy in large scale.  Changing paper

          9        things in large scale generally is very, very

         10        hard.

         11             Conversely, I am not sure that the models

         12        for protecting paper have moved forward to

         13        match the precision that is now demanded in our

         14        elections and the ability to report quickly and

         15        report accurately and the conflict that is

         16        created between the reporting of the first

         17        count as an electronic count that may not

         18        include all of the ballots, that may not have

         19        the precise construction of all of the input of

         20        the public, and then producing a second count

         21        that has additional information that may

         22        conflict with the first count.

         23             So just the damage to public confidence

         24        that that notion has is something that, I

         25        think, hasn't been addressed, and it is



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 34





          1        directly because of the emergence of the

          2        electronic aspect of voting.

          3             >> KING:  Yes?

          4             >> TOREGUS:  Just to piggyback on that, on

          5        Section 8 of the VVSG, there is a beautiful

          6        diagram, a process diagram, of what happens

          7        during an election, and stimulated by what

          8        Christopher -- if I might call you that --

          9        mentioned.  I wonder if the people who put

         10        together that flowchart were able to somehow

         11        put it side by side with what people might call

         12        the legacy process and make sure that the kinds

         13        of computer relationships and the computer

         14        standards that are now being promulgated are

         15        being based on a theoretical concept or on a

         16        concept that actually matches the reality on

         17        the ground.

         18             I am not speaking about whether the

         19        reality on the ground is good or not.  I am

         20        just saying that if it is not matching, then

         21        we're going to incur significant change

         22        management cause of every organization that

         23        conducts elections.

         24             And whereas we can easily -- let's say

         25        that -- easily change something that the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 35





          1        machine does, when we talk about thousands and

          2        thousands of institutions, we have to be a

          3        little bit more cautious.

          4             So to look back to my question, when

          5        the -- I think it was 8.1.2, when the procedure

          6        diagram was put together, was that an idealized

          7        procedure, or does that reflect the current

          8        state of practice in the field?  I would like

          9        to know that.

         10             >> KING:  Okay.  I think we're going to

         11        find out.

         12             John?

         13             >> WACK:  I don't know if you will not.

         14        First of all, thank you for calling it

         15        "beautiful," "pretty," I don't remember, but if

         16        you like them, I have some artwork myself that

         17        I could interest you in.

         18             The models were put together by

         19        researchers at NIST working with members of the

         20        TGDC, but also looking at previous models,

         21        diagrams and talking with numerous election

         22        officials who have been involved in the

         23        development of previous versions, and they

         24        simply form what is thought of as kind of the

         25        general flow of operations, general flow of



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 36





          1        what happens in various aspects of voting,

          2        pre-election and so on, so forth.

          3             And it is there sort of as a framework for

          4        the requirements.  In other words, you know,

          5        the requirements are based upon the general

          6        practice, you know, as shown by these flow

          7        diagrams.

          8             Are they up to date with existing

          9        practices?  We believe they are.  But I should

         10        caution that they are also fairly general.

         11        They don't get into too many specifics.

         12        They're intended to be kind of what you would

         13        normally expect to happen, regardless of what

         14        state or what county.

         15             And then I also wanted to piggyback myself

         16        on Alec's point, too, about just emphasizing

         17        again, that, for example, if software

         18        independence results in -- well, the VVSG, or

         19        the recommendations, I should call them --

         20        refer to it as an independent verifiable

         21        record, but if it is pushing paper, that's fine

         22        and dandy, but an assessment of all of this

         23        needs to ensure that paper itself is

         24        sufficiently usable to be used as an auditing

         25        tool.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 37





          1             I think that that may have got lost a

          2        little bit or maybe not addressed as well as

          3        perhaps it needs to be.  We've heard a lot of

          4        feedback from people in the election community.

          5             And essentially if it's -- if an approach

          6        like this gets used generally, it has to be

          7        extremely usable, and more work definitely

          8        could be done in that area.

          9             >> KING:  Okay.  Juan?

         10             >> GILBERT:  Yes, Alec's comments prompted

         11        me along a certain line of thinking.  I am not

         12        a risk assessment expert, but it brought up

         13        some questions when you mentioned the Brennan

         14        study and some studies you have done.

         15             One of the things, when we're talking

         16        about software independence, is there such a

         17        thing as risk assessment independent, such a

         18        way that the risk is independent of the

         19        specific voting equipment?

         20             So listening to the conversations we've

         21        had so far, it appears that -- it sounds as if

         22        the actual assessments are highly dependent on

         23        the actual equipment that is being used.

         24             So one of the things that came to my mind

         25        is this idea of classifying these threats with



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 38





          1        a different metric.

          2             So I had a question for you, Alec.  Is

          3        there a metric such as the number of ballots

          4        impacted by a threat to be used to classify

          5        threats?  So if you had a metric such as that,

          6        to me it appears that would be independent of

          7        any system.

          8             So if you had catastrophic, moderate, low,

          9        none classifications, and you had a number of

         10        ballots that were impacted by each level, if

         11        you had such a model, would that be independent

         12        of any system?  That's my question.  And does

         13        such a model exist?  That is my question.

         14             >> YASINSAC:  As I said, I am not aware of

         15        one that exists, and I did put one together for

         16        this panel, but I didn't get it in time to be

         17        distributed.

         18             And one of the attributes of the model

         19        that I constructed was the impact, and I did

         20        not go so extensive as you, but exclusively

         21        mentioned that it was wholesale or retail.

         22        Either it was a large-scale attack or a

         23        small-scale attack of a few ballots.

         24             So that is an attribute that I think is

         25        essential to a threat model.  And there are a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 39





          1        number of them, as I have been working on this,

          2        actually, for a good period of time, and

          3        interacting with a lot of different people,

          4        there were many attributes that popped out as

          5        being obvious to me that they fit in.

          6             Where that kind of model obviously gets

          7        difficult is when you get further down the tree

          8        and you need to know what level of detail is

          9        appropriate to not exclude some equipment or

         10        some types of systems, and then to take the

         11        discussion pertinent to the threats on the

         12        scale that you need to be able to discuss,

         13        whether that scale is for cost, for risk

         14        voidance at the point of the elections

         15        officials, or for consideration of developing

         16        new and innovative solutions for voting

         17        systems, how do you design that threat model.

         18             So it has to be -- but I think there are a

         19        group of attributes that are widely acceptable

         20        and that we could identify if we were to spend

         21        some time on it.

         22             >> KING:  Okay.  I wanted to add, too, I

         23        had the opportunity to see a presentation by

         24        the state election director of Louisiana on

         25        their contingency planning model, and that's an



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 40





          1        interesting model to look at in terms of

          2        realistic assessment of what can go wrong in an

          3        election in a scale that is really staggering.

          4             So perhaps in addition to the models that

          5        have been discussed, there are some things that

          6        are in circulation out there already that can

          7        be brought in.

          8             What I would like to do, if there are no

          9        more comments on this -- I am sorry.  Peter?

         10             >> RYAN:  Yes, I would like to follow up

         11        on Brian's comments about the measures, because

         12        we actually toyed with a very similar sort of

         13        measure back in Utah that was slightly a

         14        different measure of impact on the number of

         15        votes that can be changed divided by the number

         16        of people who would have to collude in order to

         17        do that.

         18             That seems to be an actual measure in some

         19        respects.  It seems to work quite well in a

         20        class of systems, but we quickly realized if

         21        you try to push it to the extreme, it starts to

         22        break up.

         23             The classic example is, if you start this,

         24        to think about the graphic schemes.  There, the

         25        idea is to achieve software independence and so



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 41





          1        on.  Perhaps the risk boils down to, is there

          2        some undetected flaw in the mathematical

          3        arguments, the logic, and crypto, and so on and

          4        so forth.

          5             So there you get experts that look in the

          6        system and judge [Indiscernible] and so then,

          7        the number of people who, in some sense, might

          8        collude in making a mistake of the system, is

          9        an entirely different kind of class of people

         10        than the class of people who might work with

         11        election officials.

         12             So you try to -- I usually try to apply

         13        this measure across the entire spectrum of

         14        system, and I don't think it works for that

         15        kind of reasoning.

         16             For example, if you are ignoring it -- the

         17        kind of person who would be involved, I hardly

         18        think that Ron would find a flaw in the system

         19        and keep it to himself or something.  The odds

         20        of that happening are astronomically lower than

         21        the a number of election officials at a polling

         22        station colluding to undermine the outcome of a

         23        polling station.  So these kind of measures are

         24        interesting, but certainly, if you want go

         25        across the entire spectrum the system, I guess



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 42





          1        it gets a bit dodgy.

          2             >> RIVEST:  I think the Brennan Center,

          3        some aspects, I think they were talking about

          4        election officials would certainly be expected

          5        much higher integrity than a typical voter, you

          6        know, where if you are doing something like

          7        chain voting where you just have to persuade a

          8        voter to turn over their vote, that would be

          9        persuading an election official to violate

         10        their oath, so that would be a much, much

         11        higher barrier.  And so I think that that kind

         12        of consideration is very relevant, yeah.

         13             >> KING:  Yes, Costis.

         14             >> TOREGAS:  One last thing on 1B or 1A,

         15        you asked how do you value what is an allowable

         16        level of risk?  It seems to me that we also

         17        have to stop and pause and think who says

         18        allowable?  And ultimately, from my experience

         19        of decades of working with government

         20        officials, ultimately the people that have to

         21        decide what is an allowable level of risk are

         22        not the so-called experts, computer scientists

         23        or even the industry, but it is the people who

         24        are actually going to buy the machines and make

         25        sure that elections run well.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 43





          1             So we have to start thinking about how do

          2        we communicate very complex issues of risk

          3        assessment and risk management to people who on

          4        a daily basis have to make very complex

          5        decisions?  Somewhere along this line we have

          6        to start thinking about communication to other

          7        sectors.  And that is something that I am very

          8        much an advocate of, instead of having sectoral

          9        dialogues like we have a computer scientist and

         10        then the advocate groups, I think we have to

         11        learn to begin to discuss things across those

         12        groups because without doing that, it is very

         13        difficult to identify questions that I have

         14        just raised such as who defines.

         15             >> KING:  Costis, do you think that is

         16        also related to something I think we have heard

         17        from two panelist about the need for an

         18        expanded lexicon and a taxonomy so that we

         19        can -- when we do share these ideas, it is an

         20        accurate sharing of the ideas?  Okay.  Mike.

         21             >> SHAMOS:  In response to the question of

         22        who defines, initially it is the state

         23        legislatures.  And typically when they have

         24        acted to define the allowable level of risk,

         25        they have defined it as zero.  There are



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 44





          1        numerous requirements in statutes in all the

          2        states that refer to things like absolute

          3        accuracy and prevent every person from

          4        interfering or tampering with the equipment.

          5        So they are phrased in absolutes.

          6             For federal -- elections for federal

          7        offices, it would be the Congress.  There is

          8        already a definition of an acceptable error

          9        rate, and it will end up being left to the VVSG

         10        to say what an allowable level of risk is

         11        because the states will inevitably accept that

         12        unless so it is out of kilter with their

         13        expectation or what is publicly acceptable.

         14             If you go up to somebody in the street and

         15        you say, what is an allowable level of risk

         16        that somebody could steal an election?  They

         17        would say zero.  We just can't have it.  That

         18        is completely unrealistic of them to say that.

         19        But it is certainly the desire.

         20             We have had a lot of useful discussion

         21        about this question.  Unfortunately, I have the

         22        feeling that we didn't answer it.  So I am not

         23        sure that what we have done is of great utility

         24        to the E.A.C.

         25             >> KING:  If you would and I have



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 45





          1        another -- I have several here on the table.

          2        Mike, I think that is going to be recurring

          3        dilemma throughout today.  But what the E.A.C.

          4        has asked of us and the Standards Board is do

          5        the best that we can, and I think along Costis'

          6        suggestion is get that dialogue going.  And I

          7        did not see the order in which things went up

          8        here, but Chris I saw yours first, so I call on

          9        you.

         10             >> THOMAS:  Just a quick comment.

         11        Addressing what the E.A.C. can do.  And one, I

         12        would give the E.A.C. credit and Brian Hancock

         13        in particular for his Denver conference that he

         14        had earlier this year that did exactly that.

         15        Bringing together the various elements.

         16             And the other thing I would point out is

         17        that the E.A.C. is moving forward with

         18        management guidelines.  And to some extent,

         19        management guidelines are analyzing risk.  In

         20        other words, how really you modify the legacy

         21        system and how you operate elections with these

         22        new systems.

         23             And this comes back to beg the question,

         24        who makes the risk analysis?  And obviously it

         25        is not any one sector.  It needs everybody to



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 46





          1        be involved in that.  And as Michael indicated,

          2        ultimately legislatures are the ones that weigh

          3        in there, and I think they would find some

          4        decent guidance coming from a federal level

          5        with input from the wider community and making

          6        this risk analysis.  As opposed to just them

          7        making the analysis, which as Michael

          8        indicated, they are rarely going to do anything

          9        more than zero.

         10             >> KING:  Okay.  Thank you.  Daniel.

         11             >> CASTRO:  In response to, you know,

         12        question that we have been talking about about

         13        what is an allowable level of risk.  I would

         14        argue that one thing is that is what states and

         15        Congress is willing to pay for.  I think when

         16        you ask the people on the street, yes, they

         17        want zero and legislators want zero risk.  But

         18        as a public policy question, this comes down to

         19        how much are you willing to spend?  And this is

         20        what Costis was talking about.

         21             I do want to go back to another point that

         22        we have talking about in terms of risk

         23        analysis, and it seems like there are some

         24        consensus at least that we -- there is no

         25        defender of risk analysis that has been done



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 47





          1        yet and more work here.

          2             When you are doing a risk analysis, there

          3        is a number of steps but among those you have

          4        prioritizing risk, identifying policies to

          5        respond to those risks and then choosing the

          6        best implementation for those policies by doing

          7        a cost benefit analysis.

          8             It seems like, and I don't want to get

          9        ahead of where the discussion is going, but if

         10        we are saying that software independence is the

         11        policy we want to choose, it seems like that is

         12        getting ahead of ourselves because we haven't

         13        completed the risk analysis yet and we haven't

         14        completed the cost benefit analysis to decide

         15        if that is the policy that we should accept.

         16             >> KING:  Okay.  Thank you.  Ron.

         17             >> RIVEST:  Yeah, I wanted to respond to a

         18        suggestion that Mike Shamos just made that the

         19        VVSG may be the place where the allowable level

         20        of risk is determined.  And then in some sense,

         21        you know, what the VVSG does is sort of set up

         22        the proposal or maybe not VVSG, would set up a

         23        framework of what can be certified.  But the

         24        actual risk that is run by any election

         25        jurisdiction, really depends on the procedures



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 48





          1        as much as anything.  What it does is provides

          2        a range of acceptable levels of risk.

          3             The question really is, does the equipment

          4        that would be certified support, you know, a

          5        level of risk controlling the risk in a way

          6        that would satisfy the potential customers, the

          7        election officials and the voting public?  You

          8        know, for example, I mean you may have election

          9        equipment which, you know, provides the option

         10        for chain voting, for the voter, if they can

         11        take the ballot outside the polling place and

         12        exchange it, that is a say procedural question.

         13             Similarly, there may be an option on the

         14        voting equipment which allows for a set up

         15        validation so you can check the software on the

         16        system is really the software that is supposed

         17        to be there.  But those are procedural choices

         18        that the jurisdiction has to find as to how to

         19        enforce voter movement or either to exercise --

         20        the question in some sense is does the VVSG,

         21        you know, provide a range of choices to the

         22        election officials that allow them to get

         23        within their comfort zone for a level risk.

         24             >> KING:  Well, thank you.  I think that

         25        is an excellent start in terms of the scope and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 49





          1        the level of participation that we are looking

          2        for in these questions.  And in order to keep

          3        on the schedule, I would like to remind the

          4        panel at the end of today, towards five

          5        o'clock, there is going to be an opportunity to

          6        each panel member to come back and address

          7        issues that either have become better formed in

          8        their mind during the panel discussion or that

          9        they really want to make sure or emphasize, and

         10        I am trying to also take notes to help with

         11        some summation of those points as a way of

         12        prompting our memories when we get to that

         13        point in it.

         14             But at this time, I would like to move to

         15        the second question.  Matt, if we could.  The

         16        2005 VVSG states one of the goals for the next

         17        iteration of the VVSG was to create performance

         18        standards that promote innovation rather than

         19        design-oriented standards that limit design

         20        choices for potential manufacturers.

         21             And Juan, I think you have agreed to open

         22        up the discussion on that topic.

         23             >> GILBERT:  As you can see, there are two

         24        direct questions underneath that initial

         25        introduction and getting right to those



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 50





          1        questions, you know, where does this document

          2        meet or exceed this goal?  Where does the

          3        document fall short?  When I read the VVSG, I

          4        think what I liked about it very much was the

          5        performance standards are clearly defined with

          6        respect to, for example, evaluation, actual

          7        metric that need to be measured and calculated

          8        and the associated benchmarks.

          9             Now, I think these are things that are

         10        good for the VVSG or any standards guidelines

         11        to specify these kinds of things.  So I applaud

         12        the VVSG for putting those in there.  Now, I

         13        know some people will question the values for

         14        those benchmarks and say, are those valid and

         15        how broadly can they be generalized?  And I

         16        have a comment on that in a second.

         17             As I address the second question up there,

         18        is the concept of software independence as

         19        defined by the TGDC recommendations too

         20        technologically restrictive?  If so how would

         21        you change it to be more expansive to include

         22        more innovation?  So these are two questions we

         23        will address.

         24             Now, I think the language in the VVSG is

         25        clear on software independence.  In fact, I



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 51





          1        could take it and give it to a nonsoftware

          2        person and say, here, read these two sentences

          3        that define software independence and they say,

          4        oh, okay, I got it.  To me, that is a test to

          5        say the language is clear.  And so that is, I

          6        think that is an accomplishment in the VVSG.

          7             Now, although I consider this to be an

          8        excellent start with clear language, the VVSG

          9        also says that it makes a statement that paper

         10        schemes are software independent as an example.

         11        And I think there are issues and questions

         12        about making such a claim, does that exclude

         13        other things or is that a strong recommendation

         14        to use paper?  Which brings on my next point,

         15        which is, getting to innovation.  Looking at

         16        the VVSG in my opinion, I think functional

         17        requirements versus hardware and software

         18        requirements are pinnacle and most important.

         19             And I think the VVSG has these functional

         20        requirements in there.  I think that promotes

         21        innovation.  There is going to be debate about

         22        the different benchmarks and metrics to say how

         23        did we come to those and do those across?  But

         24        I think the definition in particular for the

         25        innovation class gives you the opportunity to



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 52





          1        challenge some of those if you do have

          2        innovation.

          3             After a few innovations are considered, I

          4        think we will discover how well VVSG

          5        accommodates those.  But I don't think we can

          6        get into this mode of what I call analysis

          7        paralysis, where we overanalyze something to

          8        the extent where we never get anywhere and

          9        never even get a case to test the VVSG.

         10             So what I like to do is cut my comment

         11        short.  I could go on with some additional

         12        things, but I would like to get into answering

         13        those questions up there, the two bullet

         14        points.  And I think that is a good way to lead

         15        the discussion.  I turn it back to Merle.

         16             >> KING:  Thank you, Juan.  Daniel.

         17             >> CASTRO:  Yeah, I just wanted to start

         18        off by -- I mean, the question up here is I

         19        think a good question.  And it goes to the

         20        point should we have functional standards or

         21        design standards?  But just starting off and I

         22        will put this out there.  I would guess there

         23        might be some disagreement on it.

         24             I do think the software independence

         25        although it is -- I say this in my written



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 53





          1        statements, it uses functional standard

          2        language but ultimately it is really a design

          3        standard.  When you look at it, the end result

          4        of it is it is forcing voting systems to only

          5        use software independent or any voting system

          6        that is not using software.  To me, that is a

          7        design standard.  So when we are looking at

          8        this, yes, I would say that is technologically

          9        too restrictive.  Whether or not that is good

         10        or bad for security, we can discuss that.  But

         11        is it a design standard or functional standard?

         12        I would say it is definitely a design standard.

         13             >> KING:  Okay.  Thank you.  Mike.

         14             >> SHAMOS:  So I think there is a

         15        significant block of folks on the panel today

         16        who have issues with software independence as a

         17        concept.  I am one of them.  And what I want to

         18        do is raise the what I think are from my point

         19        of view the major issues.  I think software

         20        independence is unarguably wonderful, warm, and

         21        maternal.  We certainly don't want to say that

         22        it is okay for software on its own to influence

         23        the outcome of an election.  And I don't think

         24        though that by rejecting software independence

         25        we are saying that it is okay for software to



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 54





          1        do that.

          2             So it starts out with a kind of -- with a

          3        motherhood feel to it that seems hard to argue

          4        against.  It doesn't go far enough.  What

          5        software independence says is that the software

          6        itself can cause an undetected change in the

          7        outcome of an election, but it doesn't take

          8        into consideration other things that might

          9        cause an undetected change in the outcome of

         10        the election.

         11             What software independence says is you

         12        can't rely only on the software.  You are going

         13        to rely on something else.  And yet there is no

         14        discussion of the something else and its

         15        integrity, and if the something else is

         16        handling of traditional pieces of paper in the

         17        traditional way that have always been used to

         18        influence the outcome of American elections,

         19        then software independence isn't so good.

         20        Because what it is doing is turning over the

         21        responsibility to system that have known flaws,

         22        and so I think software independence, if there

         23        were another statement that said what is the

         24        other thing that we are going to rely on if we

         25        are not going to rely on the software, and we



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 55





          1        could evaluate that, then we would get

          2        somewhere.

          3             There has never been a comparative study

          4        of the security of D.R.E. systems with the

          5        security of paper-based systems.  It has never

          6        been done.  And there seems to be a belief that

          7        once you put something on paper, it either

          8        becomes sacrosanct, is unalterable, or over the

          9        centuries we have developed these excellent

         10        paper handling methods that guarantee that no

         11        human being can influence the outcome of an

         12        election.

         13             How about if we had a standard of human

         14        independence, which is so to say that no

         15        undetected human can cause an undetected change

         16        in the outcome of the election.  You wouldn't

         17        argue that either.  That is a motherhood kind

         18        of thing to say.  We know probably can't

         19        achieve that either.  And I think that what we

         20        are doing by requiring software independence is

         21        frankly I think it is a subrose way of simply

         22        mandating paper trails.

         23             Because the VVSG itself even says that the

         24        only known systems that achieve software

         25        independence are paper trail systems, and it



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 56





          1        admits begrudgingly that possibly somebody

          2        might come up with something else in the future

          3        but they are relegated to the innovation class

          4        for which we have no standards because we have

          5        never seen such a thing before because we don't

          6        think it can be done.  And I think the effect

          7        that has on potential developers of software

          8        systems is extremely negative.

          9             Also, I haven't been able to find another

         10        field of endeavor, maybe somebody can enlighten

         11        me in which software independence is a

         12        requirement.  Not in the handling of nuclear

         13        weapons, not in the handling of passenger

         14        airliners that's hold 750 people.  Not in the

         15        handling of trains and other devices in which

         16        human life is at risk.  We don't require it

         17        there.  And the reason is we don't know how to

         18        make software independent systems.  There is no

         19        book.  There is no manual.  There is no set of

         20        standards that tell you how do you make a

         21        system that is completely software independent.

         22             What are the other things that you are

         23        going to shove the responsible on and how do

         24        you make those good?  And so to show up with a

         25        set of standards that say, ahh, we have this



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 57





          1        magic thing called software independence and

          2        you have got to have it.  We can't tell you how

          3        to do it, but you have got to have that.  And

          4        oh, by the way, pieces of paper do it.  That is

          5        not hanging together for me.

          6             >> KING:  Alec.

          7             >> YASINSAC:  Stole a bunch Of my

          8        comments.

          9             >> SHAMOS:  I am sorry.  I should have let

         10        you put yours up first.

         11             >> YASINSAC:  I think there is no doubt

         12        and in fact, I think most of the folks that

         13        propose software independence would agree that

         14        the goal here -- and in fact it was in

         15        Dr. Biford's statement that he just doesn't

         16        feel like a voting system can be devised that

         17        doesn't have a paper trail.  And I think that

         18        is a fairly common opinion among a lot of the

         19        folks, in fact maybe everybody universally that

         20        supports the software independence viewpoint.

         21        And it is understandable.

         22             I certainly understand why it makes a

         23        difference, as I mentioned before.  It is

         24        clearly harder to change lots and lots and lots

         25        of paper ballots in a single stroke of the pen



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 58





          1        or single movement of the keyboard or to

          2        generate.  We can argue the merits of that at

          3        length and ad nauseam.  But what I would like

          4        to suggest and ask is part of the VVSG and a

          5        part of the consideration of moving forward,

          6        instead of arguing about whether it is a paper

          7        trail or an amendment or not, is to back up and

          8        try to describe what the properties are that we

          9        want this paper ballot to have.

         10             Does it have to be human readable?  Does

         11        it have to be immutable and in what sense does

         12        it have to be immutable?  Does it have to be

         13        storable?  Does it have to withstand heat?

         14        Does it have to be something that can be seen

         15        or something that can be read or interpreted in

         16        the disabled community without any help or

         17        without devices?  What is it about this paper

         18        trail that makes it essential to the voting

         19        process?

         20             Because none of that that I can -- I

         21        couldn't find any of that in the VVSG myself.

         22        There was a discussion, I believe, in chapter

         23        four.  But the definitions for example of IVVR

         24        and VVPT and VVR in the document were really

         25        circular and were of little to no help.  The



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 59





          1        discussion of the testing gave a better insight

          2        to the notion that software independence

          3        requires something that people can see.

          4        Something that they can feel.  What are the

          5        driving properties of that medium?

          6             >> KING:  Okay.  Thank you.  John, I think

          7        you were next.

          8             >> WACK:  Let's see.  My role is with

          9        regard to this question really is just to talk

         10        about what is in the VVSG, and I wanted to

         11        point out that I think the TGDC has the

         12        resolution for software independence in

         13        December of 2006.

         14             And so the immediate question after that

         15        was let's make sure that those systems that use

         16        paper records do so in a way that is as usable

         17        as possible or as usable as it is possible to

         18        make them in the amount of time we had

         19        available.  So the requirements for example for

         20        voter verified paper trail systems were in my

         21        opinion improved a good bit, especially in the

         22        way that paper is dealt with.  The reliability

         23        requirements were approved a great deal to

         24        ensure that failure rates of printers and

         25        accordian jams of paper and things of that sort



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 60





          1        should not happen.

          2             And then ultimately in the testing, there

          3        is a large scale volume testing approach.

          4        Where in essence there is going to be a mock

          5        election as part of the testing campaign.  And

          6        systems will be tested pretty much from the

          7        beginning to the end and end testing.  And that

          8        in itself, in my opinion will put to rest many

          9        problems with voting systems and many problems

         10        with use of paper records.

         11             So I do believe that while S.I. states a

         12        goal, that it is backed up with more attention

         13        paid to making sure that paper records are

         14        usable.  Could more work be done in that area?

         15        Definitely so.  I think that is definitely

         16        true.  Could IVVR be examined more closely and

         17        more of a performance aspect be given to the

         18        requirements.  That is definitely true.  I

         19        think though that what is in there essentially

         20        constitutes good starts in those areas.  And I

         21        would like to just make sure that people look

         22        at the area first and form conclusions

         23        afterward.

         24             >> KING:  John, if can I ask a follow-up

         25        question, and then I will go to Peter and Ron.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 61





          1        In one of the projects that preceded the NIST

          2        management of the VVSG was the IEEE P1583

          3        project.  And on that project, we struggled

          4        with the notion of functional specifications

          5        versus design specifications, curious if those

          6        kind of questions came up early in the NIST

          7        project and how those were resolved.

          8             >> WACK:  Well, they did, and they are

          9        terms of art to a certain extent, and the

         10        conclusions I, myself, drew and some others

         11        were that you could go down a rathole pretty

         12        quickly arguing about what is a design

         13        requirement, a functional requirement, what is

         14        a performance requirement.

         15             And design requirements, for example,

         16        which supposedly limit the design to a specific

         17        approach, are not necessarily bad.  Performance

         18        requirements are not necessarily good.

         19        Performance requirements, where they make

         20        sense, are good.

         21             For example, in the VVSG, there are

         22        performance requirements for usability of

         23        interfaces, and the performance aspect really

         24        is:  We don't care what it looks like, to a

         25        large extent.  We don't care how big it is, how



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 62





          1        small it is, what color it is.  We just want to

          2        make sure people vote accurately when they use

          3        it.  So you're writing to a certain performance

          4        level of accuracy.

          5             In some other areas, security is a good

          6        example.  It is not a good idea to strictly

          7        say, it must be secure.  You know, you have to

          8        get -- well, it is probably a complicated

          9        discussion, but there are a number of ways in

         10        which security controls can be implemented

         11        incorrectly, and so to a certain extent, there

         12        has to be some fairly specific design-related

         13        requirements.

         14             I would say, though, in general, we wanted

         15        to go more towards performance-based

         16        requirements, obviously because it just makes

         17        it easier for vendors to come up with good

         18        solutions.

         19             So I think that was pretty much what

         20        people wanted to do.  In areas where that

         21        didn't occur, perhaps that can be approved upon

         22        in the second draft.  But in some areas, I

         23        think it was justifiable to stick with design

         24        requirements.

         25             I did mention functional requirements.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 63





          1        Functional requirements are how a system --

          2        what a system ought do, and they're kind of

          3        in-between design and performance.  And again,

          4        I'll reiterate, you go down a rathole pretty

          5        quickly with a lot of these terms.

          6             But in general, yes, we wanted to go

          7        towards performance as much as possible.

          8             >> KING:  Thank you, John.

          9             Peter?

         10             >> RYAN:  Yes.  Well, there is quite a few

         11        issues here which need clarification.  I can't

         12        speak to the details on how things are worded

         13        in the document.  I can speak from my own point

         14        of view.

         15             To me, certainly software independence

         16        doesn't mean paper audit trail, certainly, in

         17        the sense of a VVPAT, human readable,

         18        necessarily.  That is the first comment I need

         19        to make.  If that's what the document seems to

         20        imply, that would be an unfortunate phrasing of

         21        the document.

         22             The other point is that, certainly when I

         23        came across the term first, I guess from Ron, I

         24        took it as meaning it is not the be end, the

         25        end all.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 64





          1             To answer Mike's comment, yes, I fully

          2        agree if you make the system, it's

          3        dependability is independent of the software,

          4        but you need to also investigate a lot of other

          5        things.  And to a lot of us, myself included,

          6        one of the goals we're going through is to make

          7        the particular integrity of the system

          8        independent of essentially all components of

          9        the system -- including human beings, hardware,

         10        and so on and so forth.

         11             So I viewed software independence as a

         12        sort of first step, I guess largely on the

         13        grounds -- as I think Ron pointed out -- is to

         14        make sure it is the most critical, most

         15        vulnerable piece of system, for example, D.R.E.

         16        systems.

         17             So that was my view, that was the first

         18        step, but surely, we need to go beyond that.

         19        And my final point, I think, is that you can go

         20        beyond that.

         21             And there seem to be certain comments here

         22        that systems don't exist or seem unconceivable

         23        that could achieve these goals of software

         24        independence and perhaps independence of humans

         25        and officials and so forth.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 65





          1             And I think we have a group of concepts,

          2        several of us, that say that is actually not

          3        true.  Now is not the time to talk in detail

          4        about them.  But I believe such systems do

          5        exist, at least in theory.

          6             My question is about whether if, in

          7        practice, they're pretty reliable and we have

          8        to argue about the cost of employing them, and

          9        whether they are credible in the eyes of the

         10        electorate at large, the stakeholders, and

         11        issues like that.

         12             But in principle, I think there are

         13        systems.  We have a group of concepts that

         14        something like this can exist.  I'll leave it

         15        at that.

         16             >> KING:  Ron?

         17             >> RIVEST:  Thanks.  I also wanted to

         18        respond to some of the points that Mike Shamos

         19        raised with respect to software independence.

         20             You brought the airplane analogy, which is

         21        one that is commonly brought up, and I think it

         22        is an interesting one to sort through and to

         23        think about.

         24             It is actually an interesting one for lots

         25        of reasons.  An airplane is actually the prime



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 66





          1        example of multiple redundant systems and

          2        backup systems, which is really what we're

          3        talking about with software independence here,

          4        is having another way of being able to achieve

          5        a vote count.

          6             They've got multiple hydraulic systems.

          7        If one of them goes down, you have another one

          8        there.  The pilot has a separate oxygen tank

          9        and oxygen mask, in case there should be

         10        decompression, you know, so that he can fly the

         11        plane even in spite the failure of a

         12        compression system.

         13             There is even talk about the plane being

         14        able to land itself if it should be overcome by

         15        a terrorist and such.  So there is lots of

         16        mechanisms for protecting against failure of

         17        any one given component.

         18             But even more so, the airplane analogy

         19        doesn't go far enough because, when you think

         20        about it hard, voting systems have a higher

         21        calling, if you will, than an airplane does.

         22        An airplane, when it functions correctly, gets

         23        the passengers from point A to point B.  When

         24        you are there, you know you're there, and you

         25        know you're in the right town.  Although when I



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 67





          1        landed in Austin last night, the taxi driver

          2        said, Do you know where you are?  I said, No.

          3        He said, You are in London.  This is London,

          4        England weather.  But you know where you are

          5        when you get to the airport.

          6             So the correctness of the operation of an

          7        airplane is sort of self-evident when you land

          8        at the airport.  The correctness of the voting

          9        system is not so self-evident.

         10             A winner is announced, and you wonder, is

         11        that really the correct winner.  It is not as

         12        obvious as being at the airport in Austin,

         13        Texas.

         14             So I think a voting system has to meet a

         15        higher calling.  It has to be not only correct,

         16        but demonstrably correct.  You have to be able

         17        to demonstrate to everyone, the voters and the

         18        losers, that you got the right answer, in a way

         19        that, merely pointing at the software and

         20        saying, this software is certified and we think

         21        the software in the machines is the same

         22        software that is certified and there has been

         23        no viruses, etcetera, probably does not suffice

         24        for a lot of people.

         25             So the difference between being correct



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 68





          1        and demonstrably correct I think is part of the

          2        point.  It certainly is possible and maybe not

          3        even hard to write D.R.E.'s that are correct.

          4        I think we realized that early on in the work

          5        at the TGDC.

          6             The hard part with the voting system that

          7        is all software is to make the results

          8        demonstrably correct.  How do you persuade

          9        somebody that it is actually giving you the

         10        right answers?  And that is a near impossible

         11        task.

         12             >> KING:  Thank you, Ron.

         13             Juan, you next, and then back to Mike.

         14             I was struck by something you said, Ron,

         15        about voting systems having a higher calling,

         16        and I think all of us in the room agree to

         17        that, but I am not sure where the charter is

         18        for that.

         19             Because we talked earlier today about, in

         20        risk assessment, that normally is something

         21        that is done off of a collection of priorities

         22        established by the organization, and I don't

         23        disagree with what you said.

         24             But as we start to look back for the

         25        artifacts that prove that or attest to that,



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 69





          1        that could be very helpful in establishing some

          2        of our risk models.

          3             Juan?

          4             >> GILBERT:  I think the gentleman --

          5             >> KING:  The gentleman from Auburn is --

          6             >> SHAMOS:  Yes, he wants the last word.

          7             So as far as airplanes, yes, I agree, that

          8        the analogy is certainly not perfect.

          9             But I was struck by something you said,

         10        which I liked, and I am going to throw it back

         11        to you to confirm that you really said it,

         12        which is that redundant systems on the airplane

         13        are sufficient.

         14             If that is true, then if we had redundant

         15        software systems in a voting machine and the

         16        two were really independent of one another,

         17        they were different code bases, came from

         18        different places, etcetera, it seems to me that

         19        ought to be able to satisfy the requirement of

         20        software independence; that is, this other

         21        thing that we're talking about, that is going

         22        to take the reliance on software A could be

         23        software B.  I don't think you're going to say

         24        yes to that.

         25             >> RIVEST:  Can I answer that, please?



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 70





          1             >> KING:  If it is a short answer.

          2             >> RIVEST:  It is not a short answer.

          3             >> KING:  Answer, anyway.

          4             >> RIVEST:  So the -- that's a path which

          5        the TGDC explored at length.  I mean, you're

          6        following down the footsteps of thinking that

          7        the TGDC went when it was involving the VVSG.

          8             It seemed very attractive at first, to

          9        take that analogy of redundant systems in

         10        looking at redundant software systems to try to

         11        provide that confidence that you want.

         12             In the end, there are no marketplace

         13        examples of that, first of all.  And in the

         14        end, it seemed that that was actually an

         15        illusion.  You were chasing after a mirage most

         16        likely.

         17             It may turn out to be workable in the end,

         18        but I don't think it is likely to happen real

         19        soon for the following reason:  Because

         20        independence is an illusion, I think, when you

         21        try to build such a system.

         22             How do you build such a system?  You have

         23        vendor A producing part A.  Vendor B producing

         24        part B.  The election officials need to choose

         25        it.  Well, they need to choose some system



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 71





          1        integrator to put those parts together.  The

          2        system integrator is looking for the code of

          3        part A and part B.  The independence is lost at

          4        the point you're putting it together.

          5             And so I think the idea that you can

          6        actually combine disparate systems in a way

          7        that preserves what you want.  The

          8        independence, is extremely tough to do in

          9        practice, because you would have middlemen

         10        stepping in right away to provide lack of

         11        independence, to provide a single point of

         12        accountability for the election officials

         13        should something go wrong, and then they will

         14        be able to play with either system and change

         15        the outcome.

         16             Second of all, you have the issue of being

         17        demonstrably correct as well, which you still

         18        have two software systems which are

         19        complicated, and then trying to argue that

         20        they're both correct.

         21             It is very common with inversion

         22        programming to see programmers making the same

         23        errors.  It happens all the time when people do

         24        studies.  So it is not any way a guarantee that

         25        you're getting the independence you want.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 72





          1             And finally, to answer the question, where

          2        does the charter come from for demonstrably

          3        correct, I think the legitimacy of the

          4        government, the elected officials, the voters

          5        want to see depends on the confidence of the

          6        election.  That is where the charter comes

          7        from, is being able to demonstrate in a

          8        convincing way to the voters and to the losers

          9        that the outcome is correct.

         10             >> SHAMOS:  Okay.  Thank you.  I had

         11        yielded temporarily.  I want to finish up my

         12        comments now.

         13             >> KING:  Go ahead, Mike.

         14             >> SHAMOS:  I don't know if it would be a

         15        surprise to anybody or not, but there is no

         16        D.R.E. with VVPAT that exhibits software

         17        independence as it's defined in the VVSG.

         18             And if those systems were submitted to a

         19        voting system testing laboratory today, and the

         20        testing laboratory did its job, it would not

         21        pass them.

         22             And the reason for that is that all of the

         23        VVPAT systems -- I am not talking Op Scan --

         24        all of the VVPAT systems, the VVPAT itself is

         25        created by software.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 73





          1             And in all of these systems, there is some

          2        code or some unreadable device that the human

          3        cannot interpret that tells whether or not his

          4        ballot is valid.  And so if he can't tell that

          5        a valid ballot has been printed out, then it is

          6        not independent of software.

          7             Furthermore, the idea that there is --

          8        we're after this demonstrability, which I would

          9        love to achieve, but how can it be that having

         10        pieces of paper in boxes and rolls, which are

         11        handled by many humans and counted erroneously

         12        by any humans provide any kind of

         13        demonstrability that the totals in the election

         14        were correct.  They haven't since the 1850's.

         15        I can't imagine what has happened in the year

         16        2007 to solve that problem suddenly.

         17             I agree that, to a certain extent, Op Scan

         18        ballots achieve software independence to a

         19        certain degree, at least to the extent that the

         20        ballot is not created by a computer and,

         21        therefore, is independent of software.  I'll

         22        agree with that, but it still has all of the

         23        other problems.

         24             So what I think, if we're talking about

         25        ghosts, chasing software independence as a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 74





          1        necessary aspect of a system for voting but not

          2        a necessary aspect of a system for anything

          3        else that we do, it sets an unnecessarily high

          4        standard that rules out all kinds of wonderful

          5        systems, including systems that have been fully

          6        operational for 25 years, all of a sudden they

          7        become unacceptable, because suddenly, we can't

          8        demonstrate that they work.

          9             I don't agree that we can't demonstrate

         10        that they work.  We may not be demonstrating it

         11        to a certainty.  But certainly, looking at

         12        appropriate testing protocols for things, it

         13        gets us to a sufficient level of certainty that

         14        the system has been working.

         15             It always has.  It does in everything else

         16        that we use in life.  And so I don't understand

         17        why we have to produce 100 percent proofs of

         18        correctness for everything that is going on in

         19        an election.

         20             >> KING:  Thank you, Mike.

         21             Peter?

         22             >> RYAN:  I would like to pick up again on

         23        this sort of avionics analogy, because first of

         24        all, to add to Ron's comment, the avionics

         25        failure is a manifest; whereas, there is no



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 75





          1        God's eye view of what the outcome of an

          2        election could be.  We can't just do an

          3        extrinsic check to see that the outcome is

          4        correct.  That is one fundamental difference.

          5             The other difference is how we can

          6        recover.

          7             >> AUDIENCE MEMBER:  Microphone.

          8             >> RYAN:  Sorry.  Is the issue about how

          9        you recover if you detect errors.  If avionics

         10        goes critically wrong, you don't recover, and

         11        the plane crashes.

         12             With the kind of software independence

         13        systems that some of us are investigating with

         14        voting systems, if you detect an error, you can

         15        potentially recover from it if you've got a

         16        suitable recovery mechanism.  So I think that

         17        is another fundamental difference.

         18             And so another reason why you need triple

         19        redundancy and so forth in avionics, is that

         20        you're going to have to go on the possibility

         21        that if things go wrong, you may not have a

         22        good recovery mechanism.

         23             And by definition, if we're going to have

         24        proper avionics, the system is going to have to

         25        be software dependent; whereas, I think we



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 76





          1        have -- as I mentioned earlier -- proof of

          2        concept demonstrations with voting systems,

          3        that we can have software independent systems,

          4        so there are all bunch of systems in which I

          5        think the avionics analogy doesn't really hold

          6        water.

          7             Yes, I'll oppose that.

          8             >> SHAMOS:  Well, if I can respond for a

          9        second.  Give me 15 seconds.

         10             >> KING:  Go ahead.

         11             >> SHAMOS:  I actually agree that in

         12        avionic systems, that failures manifest.  Yes,

         13        it is true.  It is often if a plane crashes in

         14        the place where we can see it, then we see the

         15        wreckage.

         16             But very often, the NTSB is completely

         17        unable, after years of study, to figure out

         18        exactly what it was that caused the plane to go

         19        down, whether it was a software problem or

         20        something else.

         21             By the way, I am not arguing against

         22        redundancy or cryptographic systems.  I was a

         23        big supporter of both here.  I am with you on

         24        all of these things.  What I don't understand

         25        is the requirement of software independence.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 77





          1             If you can achieve it, it is a good thing,

          2        but requiring it of all systems is what seems

          3        to me to be unnecessarily restrictive.

          4             >> KING:  Thank you.  Ron and then Peter

          5        and then we will wrap up.

          6             >> RIVEST:  I was just puzzled by again

          7        Mike's comment that D.R.E.s with VVPATs are not

          8        software dependent.  I am wondering if you can

          9        sort of carefully delineate how an undetected

         10        able software bug could cause an undetectable

         11        change in the election outcome.

         12             >> SHAMOS:  It is easy.  I will take a

         13        continuous rule VVPAT.  The continuous rule of

         14        VVPAT.  Every ballot image -- excuse me, cast

         15        vote record.  Every cast vote record, there is

         16        a code printed with it so that an association

         17        can be made between that ballot image and the

         18        ballot image ostensibly stored in electronic

         19        memory so they can be reconciled.  So what I

         20        do, if I have access to the software and I can

         21        be an intruder, an undetected intruder, what I

         22        do is printout a beautiful VVPAT for the voter

         23        that has identically his choices on the VVPAT.

         24        But the code that I print is an invalid code.

         25        Then when he leaves the ballot, when he leaves



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 78





          1        the booth, I then printout another VVPAT that

          2        has the votes my software wants to it have and

          3        yet prints a code that says this is a valid

          4        ballot.  I mark it as having been spoiled, for

          5        example.

          6             >> RIVEST:  I would argue that that is not

          7        a voter verified system.  You have codes that

          8        unreadable by the voter.

          9             >> SHAMOS:  They are no voter verified

         10        systems because they all have unreadable codes

         11        on them.

         12             >> RIVEST:  You can write valid and

         13        invalid in English and then have the voter --

         14             >> SHAMOS:  Yeah, you can do that, but

         15        there are no systems in which everything on the

         16        ballot that can be used to invalidate the

         17        ballot is visible and readable to the human

         18        being.

         19             >> RIVEST:  I agree -- I am in agreement

         20        that system that does not have all of the

         21        important information that could validate be

         22        voter verifiable would not be software

         23        independent, that's correct.  So systems.

         24             >> KING:  Let the record show that we have

         25        an agreement.  Thank you.  Let's see, I don't



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 79





          1        know who was up first, Alec or John?  Oh, I am

          2        sorry.  Sarah.

          3             >> JOHNSON:  I have an easy question for

          4        Peter.  Specifically you had mentioned that you

          5        have got some systems, proof of concept systems

          6        that you are looking at, and obviously a lot of

          7        this discussion is centering with D.R.E. That

          8        is what we are talking a lot about.  There are

          9        thousands used and continuing to be used as a

         10        speak in elections.  Do you have the system

         11        that you are looking at in England or other

         12        entities, do you have D.R.E.s that meet the

         13        type of software, the two different softwares

         14        in one system that Michael was talking about?

         15             >> RYAN:  I think the first response is

         16        the kind of system I primarily look at is not

         17        really a D.R.E. system in that sense.  It is

         18        cryptographic, but it doesn't involve the voter

         19        interacting with the D.R.E. touch screen or

         20        anything like that.  So I don't know, does that

         21        answer your question?

         22             >> JOHNSON:  The question was a general

         23        question in general because we have got, do you

         24        have a system that can do this?  We have got,

         25        no, you can't get it done.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 80





          1             My question was in general about your

          2        proof of concepts system that you have

          3        mentioned.  Without going into a lot of detail,

          4        are they anything that we have seen today?  Are

          5        they Op Scan?  Are they D.R.E.s or are they

          6        something that is a hybrid that is something

          7        totally different?

          8             >> RYAN:  They are not D.R.E.s as a said.

          9        Yes, they are not.  Keep it simple like that.

         10        I guess best described it completely

         11        differently.  They are essentially

         12        cryptographic.  They do involve paper trail of

         13        sorts in the sense that the voter does -- can

         14        in principle take away, well, I think Ron calls

         15        it protected ballot form which holds their vote

         16        in encrypted form on the form.  And they can

         17        later check that that gets into a tabulation

         18        process because a bit of mathematics going on

         19        there.

         20             So there is a sort of paper trail in some

         21        sense but it is not, as it stands, a human

         22        readable at least at the point that it becomes

         23        the protected ballot.  So human readable at the

         24        time that the voter casts the vote in ways I

         25        could explain later.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 81





          1             So again we come back to this issue that

          2        there seems to be some confusion as to whether

          3        software independence is synonymous with, well,

          4        VVPAT.  Seems to be the implication.  In my

          5        mind that is clearly not true, in the sense

          6        that I understand VVPAT.  So yeah, but it is

          7        still quite theoretical.  We have prototypes of

          8        it, and we have run a trial small scale student

          9        election trial with it last year.  We will run

         10        more next year so the thing has been tested

         11        out.  Does that answer?

         12             >> KING:  Okay.  As we get ready to wind

         13        up this topic, I have got Alec, John, and the

         14        Juan.  And if you could hold your comments to

         15        may be a minute.

         16             >> YASINSAC:  Less than that.  I had

         17        proposed initially that we talk about the

         18        properties that this persistent ballot need to

         19        have, the VVR, the IVVR.  My contention is if

         20        you defined those properties and you defined

         21        them very explicitly in terms of what you

         22        needed to have to be able have a verifiable

         23        system, paper wouldn't meet it.

         24             That is my contention, and I think that

         25        has been borne out of why we are where we are.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 82





          1             But since John is next, I will mention

          2        this as well, that software independence leans

          3        heavily on voter verification.  Evidence is

          4        pretty good that voters don't verify their

          5        verifiable record even when they mark it

          6        themselves.  It is debatable about how careful

          7        the average voter is in making their own

          8        marking.

          9             So in lieu of that information, if voters

         10        don't verify the VVPAT and voters aren't across

         11        the board careful in their marking of the

         12        individual ballot then where does that leave

         13        software independence?

         14             >> KING:  Thank you.  John.

         15             >> WACK:  Just as a quick response, I

         16        don't know the percentage of voters who vote on

         17        VVPAT systems.  I believe Op Scan in the last

         18        general election was 49 percent.  And so I

         19        would just say there is a tendency to associate

         20        VVPAT with software independence, and I would

         21        sort of try to step back with from that.  The

         22        VVPAT systems that were implemented to begin

         23        with I don't think were the best design and

         24        focusing on those tends to cloud the issue a

         25        little bit.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 83





          1             The other thing I wanted to point out that

          2        was something to Mike and Ron is that the TGDC

          3        hotly debated whether there could be anything

          4        on the paper record that essentially the voter

          5        could not verify or might be a secret vote that

          6        could be somehow or another changed in the

          7        electronic memory so that it would invalidate

          8        the paper record.  VVSG 2005 it had a

          9        requirement that said that the paper record

         10        ought to have a code or shall have a code on it

         11        that can link to its electronic counterpart.

         12        Some states require that.

         13             The current recommendations basically said

         14        that has to be provided as an option.  You have

         15        to be able to turn it off because some states

         16        regard that as a violation of privacy.  There

         17        was also a lot of debate about whether there

         18        out to be any bar codes on paper record because

         19        basically a voter can't verify a bar code.

         20             And the decision was that all paper

         21        records shall be produced in a way that they

         22        can be Optically Scanned which does not require

         23        a bar code.  However, bar codes are allowed to

         24        be used most likely for the purposes of adding

         25        additional content to the paper record, for



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 84





          1        example, if the paper record is going to be

          2        scanned and read back to a blind voter, perhaps

          3        the bar code could contain pronunciation answer

          4        keys to candidates names and things of sort.

          5        But anyway, the requirement for linking the

          6        paper record to the electronic record was

          7        changed from VVSG 2005 from a I think should be

          8        provided as an option.

          9             >> KING:  Okay.  Thank you and finally,

         10        Juan.

         11             >> GILBERT:  My comments are to summarize

         12        kind of these questions.  I guess listening to

         13        Michael and Ron, one of the things that I

         14        clearly observe, and I like to bring this up

         15        and bring Brian in on my summary, which is, Ron

         16        said to his knowledge there is no software

         17        independence systems that use software as a

         18        back up or redundant observer.  I think that

         19        there are examples of that, you know, there is

         20        one that we have worked on that uses video as a

         21        verification mechanism.  And a novel way.

         22             Other people tried it but they didn't use

         23        it a certain way.  We hear Peter talks about

         24        one way to do software independence as well.

         25        As in the spirit of the VVSG and in the spirit



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 85





          1        of the things we are discussing, we constantly

          2        talk about if you have something, we need to

          3        test it.  We need the standard to test by.  I

          4        think way have this innovation class which is

          5        outlined, but this is where Brian comes in.  I

          6        think we should have some things to test

          7        innovation class itself.

          8             There clearly are some things out there

          9        that people have experimented with, that people

         10        are trying, and I think these things will be

         11        excellent candidates to give the innovation

         12        class a trial.  Looking at the definition of

         13        software independence, and straight from the

         14        VVSG, it clearly says that the voting systems

         15        software is not capable.  So that begs the

         16        question is where is the boundary of the voting

         17        system software?  So if I have one piece of

         18        software that in my mind, that is the voting

         19        system that is doing the tallying, that is

         20        presenting the ballot, doing those things.

         21             If I have an observer, that is a piece of

         22        software.  Now the argument becomes is that

         23        part of the voting system?  Or is it an

         24        independent observer?  So these are all

         25        questions I think that have to be answered, and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 86





          1        I think taking -- and this is just my

          2        recommendation, taking a few of these

          3        innovations that claim to be software

          4        independent and trying the innovation class,

          5        let's see if the innovation class will come out

          6        by trying in, quote, unquote, innovations and

          7        see if they meet these definitions.  And so I

          8        end there.

          9             >> KING:  Okay.  Thank you.  I think I am

         10        going to exercise my privilege as moderator and

         11        we are going to take a break a little earlier

         12        than planned.  I am starting to see fatigue or

         13        low blood sugar set in.  But let's --

         14             >> RYAN:  Or jet lag.

         15             >> KING:  Or jet lag.  Yes, Peter came a

         16        long way.  Let's take a 15-minute break now,

         17        and when we come back, Alec, we will start with

         18        question number three and your lead in to that

         19        question.  Thank you.

         20             (Break)

         21             >> KING:  I think we are ready to

         22        reconvene if the folks in the lobby could come

         23        in and find a seat.  We are missing two members

         24        of the panel.  Brian, did you see Sarah and

         25        Chris?



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 87





          1             >> HANCOCK:  Yes, let me see if I can go

          2        round them up real quick.  Merle, also they are

          3        working on cooling the room off a little bit

          4        since it is warm.

          5             >> Alec mentioned that he revised his

          6        documents.  I am just wondering if those

          7        documents are out on the table, his revised

          8        versions?

          9             >> YASINSAC:  They are early versions.

         10             >> KING:  Carrie, you raise a good point.

         11        Maybe at the end of today, if anybody has

         12        updates to any documents they have submitted,

         13        they can identify that.  Matt just informed me

         14        the other panelists are on their way, but they

         15        have given us permission to start without them.

         16             We are on to the third question in the set

         17        of questions that we were asked to address.  I

         18        believe that question is displayed on the

         19        screen behind us.  "Do methodologies exist to

         20        test voting system software so it can be

         21        reliably demonstrated to operate correctly?

         22        And then what added security benefits are

         23        created by S.I. that are not met by the testing

         24        process?"  And Alec has volunteered to open

         25        with some guiding remarks on that.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 88





          1             >> YASINSAC:  I would actually like to

          2        take this two directions.  The first is to

          3        address the first part of that directly, do

          4        methodologies exist to test voting system?  I

          5        think the answer to that fairly much an

          6        unequivocal no, there aren't ways to test the

          7        system to verify that it is absolutely correct

          8        and has proven that year after year after year.

          9             On the other hand, software engineering

         10        has matured to the level that we are able to

         11        provide or the software engineering is able to

         12        provide high quality software based on mature

         13        software engineering processes, implementation

         14        of best practices, keeping of data that allow

         15        processes to be modified, to be able to repeat

         16        success and not repeat failure.

         17             And I personally believe that this is the

         18        biggest omission, and it is not addressed at

         19        all in the VVSG.  It doesn't mention any

         20        reference to the advances in software

         21        engineering to be able to leverage process

         22        maturity, be able to try and achieve the level

         23        of quality that would be incorporated into the

         24        voting systems that we need.  And I will state

         25        strongly that any software that is involved in



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 89





          1        a electronic voting system needs to be high

          2        assurance.  It needs be engineered with rigor

          3        and it needs to be a software that has a lot of

          4        attention given to it to make it happen.

          5             The second place that I would like to take

          6        this is actually to address something that Ron

          7        mentioned a few minutes ago about redundancy.

          8        It is actually a little bit of comparing apples

          9        and oranges of S.I. and testing processes.

         10        Because software independence really is not a

         11        testing approach.  It is not a development

         12        approach.

         13             It really is an architecture and that

         14        architecture that provides the ability to have

         15        redundant mechanisms that can verify one

         16        another.  And in this case, it is the mechanism

         17        that is the main parachute we call it, or I

         18        call it, is the electronic system and the back

         19        up parachute is the paper trail.  And so that

         20        structure is inherently better in some ways

         21        than having an independent mechanism by itself

         22        that you try to engineer to a very, very high

         23        level of sophistication or very, very high

         24        level of assurance.

         25             So what software independence has done is



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 90





          1        provide this structure that has a bit more

          2        inherent security.  Now, it is not without

          3        problems, and that redundancy specifically --

          4        you have to think about the failure modes that

          5        those redundant mechanisms may take.

          6             For example, in the case of the paper

          7        failure, in software independent systems, there

          8        is no redundancy.  If the paper system itself

          9        is wrong at the beginning or is manipulated

         10        before the election, so that we, for example,

         11        reverse candidates on some ballots or if the

         12        paper is manipulated during the voting process,

         13        or if the paper is manipulated after the voting

         14        process, there is no redundant mechanism, in

         15        general, that can generally detect that

         16        failure.

         17             It is the redundancy of software

         18        independence, is exclusively the other

         19        direction.  It will allow you to detect faults

         20        or flaws that are on the electronic side, but

         21        it doesn't really give you the redundancy that

         22        you would like to have on the paper side.

         23             So on the flip side, if you do have

         24        electronic failure, then your failure mode

         25        allows you to detect some problems that you may



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 91





          1        have in the election, and as have been pointed

          2        out, you may or you may not be able to trace

          3        that failure back to its cause.

          4             And as was pointed out, if you can't trace

          5        it back to its cause, then in some sense, at

          6        least, the election fails, because one of the

          7        goals of the election system is not just that

          8        it be accurate but that it be demonstrably

          9        accurate.

         10             So if you have a failure, that you can

         11        detect that failure after the fact, and even --

         12        I would contend, that even if you can make a

         13        strong argument about why the result was wrong,

         14        that first result being wrong is reasonable

         15        doubt to the candidate that lost and the folks

         16        that supported the candidate that lost.

         17             So what software independence -- one of

         18        the things that it really does programmatically

         19        inject into the election system is the notion

         20        that that first count, if it is not perfect,

         21        that is okay, because we can fix it with a

         22        paper trail.

         23             I think that is the wrong approach to take

         24        for election systems.  If we have a paper

         25        trail -- and I am not against the paper trail.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 92





          1        The point is, if we're going to use it, we need

          2        to use it during the election period to have a

          3        balance of mechanisms that -- that redundantly

          4        verify one another.

          5             So that when we produce that election

          6        result on election night, it is verified, and

          7        it's correct, and it takes overwhelming

          8        evidence after the election is reported that

          9        there was fraud or whatever it would be to

         10        overturn that result, and that should be the

         11        focus, and that doesn't appear to me to be the

         12        focus of software independence.  Yes, I believe

         13        that -- yeah.

         14             I did also mention that the part of the

         15        redundant mechanism of software independence is

         16        audits, and audits, as was mentioned, they

         17        can't be required in the VVSG, although

         18        responsible election officials will do them.

         19             I am not certain that it is possible for

         20        them to be conducted with enough rigger that

         21        this redundant mechanism is going to be as

         22        effective, as many people believe it will be

         23        done.  So it certainly raises questions about

         24        the comprehensive nature of the redundant

         25        mechanisms.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 93





          1             And I, for one, believe that we should

          2        have redundant mechanisms involved to be able

          3        to verify these systems.  I am just not sure

          4        that software independence provides it and it

          5        goes far enough from my perspective.  So that

          6        is my opening comments.

          7             >> KING:  Okay.  Thank you, Alec.

          8             John?

          9             >> WACK:  If I can respond quickly in

         10        agreeing with many of the things you said.  I

         11        recalled that there was debate during the

         12        development of the requirements for IVVR as to

         13        whether they were perhaps too design specific,

         14        did they -- did they basically mandate a system

         15        that you're going to do the auditing post

         16        election.  You couldn't actually do it during

         17        the day so that, by the end of the day, the

         18        records could be -- the IVVR records could also

         19        be consulted and you could end up with

         20        verifiable results at the end of the day.

         21             And this, as well as some other concepts,

         22        were debated and I think the general consensus

         23        of the TGDC was that we couldn't -- they

         24        couldn't write requirements yet to that -- to

         25        those sorts of approaches, and so that's, in a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 94





          1        sense, how the innovation class came about.

          2             And so I -- I guess basically I am just

          3        saying that was considered, but then we had to

          4        get back to the real problem of writing a

          5        standard that is specific that doesn't

          6        constrain approaches too much, that -- you

          7        know, that people can actually write tests.

          8             And there was a constant tension, in a

          9        sense, between what we would like, you know,

         10        versus what we know how to do.  Some of these

         11        approaches were considered, but again, that is

         12        where the innovation class came in at that

         13        point.

         14             >> KING:  Alec, I had a question I wanted

         15        to follow on, looking back at my notes.

         16             You said that if paper was to be used --

         17        and I am paraphrasing, so correct me, that it

         18        should be used during the election.  Do you

         19        recall your comment on that?

         20             >> YASINSAC:  Yes.

         21             >> KING:  If you could, amplify that a

         22        little bit.

         23             >> YASINSAC:  Well, it is the notion of a

         24        true parallel test.  That notion being that,

         25        while you're conducting the election, while the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 95





          1        election is ongoing, you select random machines

          2        and you pull them offline.

          3             And you use the paper trail -- you print

          4        the paper or if it is a paper printing machine

          5        or if it's a ballot marking device, you take it

          6        offline and you run it in essentially a

          7        realtime test of what the machine is doing.

          8        And then you analyze the results of that test

          9        of the paper against the electronic record that

         10        is created.

         11             And you can detect -- in many cases,

         12        statistically, as Dr. Rivest says, you can

         13        detect, if there is an inherent fault in the

         14        software or potentially if there is malicious

         15        fault by using parallel tests during the voting

         16        day, as opposed to waiting until after the

         17        election is over to conduct audits to test the

         18        machines that have been used.

         19             >> KING:  Okay.  Ron?

         20             >> RIVEST:  I just wanted to follow up

         21        with a question to Alec.

         22             I like your concern for high-quality

         23        software engineering.  I think that would be

         24        wonderful to try to improve the current state

         25        of voting systems by getting better quality



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 96





          1        software engineering.

          2             I am just wondering if you can say a

          3        little more about how you do that and how do

          4        you set up a certification system.  When does a

          5        system fail because it has got inadequate

          6        software engineering standards somehow?

          7             >> YASINSAC:  The approach that I am

          8        advocating here is not to fail the product.

          9        What I am suggesting here is that we evaluate

         10        the process, which, again, is the way it

         11        appears to me that the industry is going, that

         12        you have a maturity level of your processes

         13        that is analyzed, via a standards body that

         14        some that are already in existence, to be able

         15        to determine how your organization produces

         16        software.

         17             If they exercise best practices, what the

         18        history of their development is.  And we'll

         19        get, I guess, into OEVT later, but the OEVT can

         20        be added in here to be a cross-check in the

         21        VVSG process to validate and verify, I guess

         22        you can say, the level -- the maturity level

         23        that has been assigned to an organization.

         24             And the OEVT will be able to give you a

         25        good indication of whether that organization



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 97





          1        has quality software processes underlying the

          2        product that they produced.

          3             It won't tell you exactly, but it gives

          4        you a pretty good idea.  Again, saying this

          5        from having just done several of these things,

          6        you can tell a whole lot about the product --

          7        the process by looking at the product.

          8             If I might, while I have my microphone on,

          9        come back.  I heard the discussion about not

         10        knowing how to write requirements for redundant

         11        mechanisms.  I know it is difficult and I know

         12        it is not probably in the literature now on how

         13        to do that.  But I would contend that we also

         14        just don't know very much about how to write

         15        requirements for software independent systems,

         16        as I have raised before.  In the VVSG as it

         17        stands now, there is not a mechanism in place

         18        that I can see that would allow me to take a

         19        system and be able run it against a processor,

         20        an algorithm, or an analysis and determine if

         21        it is -- if I present a system and say it is

         22        VVPAT, how does it become verified by the ITA?

         23        That it is actually software independent.

         24             I think it is an objective process at this

         25        point.  My point is, if the notion is we don't



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 98





          1        have a good way to write requirements for

          2        independent mechanisms, that wasn't -- didn't

          3        appear to be anything that stopped the

          4        requirement to go to software independence.

          5             >> KING:  Thank you.  Peter.

          6             >> RYAN:  Place the mic.  Is that better?

          7        So one important point which I think is maybe

          8        implicit in some of what we have been saying

          9        is, as you all gathered I am a fan of software

         10        independence, in the sense we have explained.

         11        That doesn't mean that trying to guarantee high

         12        quality of the software that runs the election

         13        isn't an issue.  Clearly we must be able to

         14        guarantee that the election will run smoothly

         15        on the day and techniques for robustness are

         16        still important.

         17             We run it -- potentially there is a

         18        paradox to the shear transparency of the system

         19        that I have been trying to and software

         20        independent systems may act against it because

         21        the whole point is to try and detect any error

         22        or corruption that occurs and be able to

         23        correct it and so on.  But of course the fact

         24        that it is done in a transparent public way may

         25        paradoxically undermine the trustworthiness of



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                 99





          1        the system.  That is something we have to be

          2        very careful about.

          3             So I want to stress this point that whilst

          4        I think as a design principle, the architecture

          5        of these systems, software independence is

          6        crucial, but that doesn't mean we ignore the

          7        quality and robustness of the software.  So I

          8        think that is one important thing.  I can jump

          9        to your other point about how we might I guess

         10        specify the property of software independence.

         11        That actually strikes me that that is one of

         12        the easier things to do here at least in

         13        theory.  And I think some of us could sit

         14        around fairly quickly come up with a full

         15        definition of what that means.

         16             We compare to that some of the other tasks

         17        and challenges we talked about earlier about

         18        how do you set up a threat model and guarantee

         19        its completeness and so on and so forth.  There

         20        are really major research problems and wide

         21        open issues in the security committee for

         22        decades and not ones we will solve in a hurry.

         23        But that one I think is actually one that is

         24        definitely doable.

         25             >> KING:  Okay.  Thank you.  John and then



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                100





          1        Costis.

          2             >> WACK:  Just quickly, excuse me,

          3        responding to Alec.  But in general, one of the

          4        problems with the VVSG is 600 pages and

          5        sometimes difficult 106, even if you have an

          6        active part in developing it.  But I contend

          7        that it does basically quantify what software

          8        independence is and make it pretty clear

          9        whether the system you are bringing for testing

         10        is software independent or not.  There is a

         11        very important chapter in there called the

         12        conformance clause and it is not a clause.  It

         13        is an actual big chapter.  But it pretty well

         14        defines the sorts of systems that can be

         15        software independent as those that use

         16        independent voter verifiable records.

         17             And currently, those would be Op Scan or

         18        VVPAT.  It pretty wall lays that out.  There is

         19        a complicated, it looks like an Alexander

         20        Caulder mobile of bubbles called the device --

         21        the class structure, the device class structure

         22        and it codifies this and it gets back to

         23        another point and that is that one of the

         24        biggest improvements that the VVSG represents

         25        is that it is made some advances on being



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                101





          1        precise basically.

          2             And one of big issues in its development

          3        was making sure that test labs and vendors

          4        agreed on what the system requirements are and

          5        how it ought to be tested.  So that forced

          6        essentially writing to what people well

          7        understood and to a certain extent, that is,

          8        you know, that has probably caused some issues

          9        in that, you know, writing more

         10        performance-based requirements where things

         11        would be more difficult to test and would

         12        require more interpretation down the road.

         13        Might have been desirable in terms of making

         14        VVSG -- what am I saying here?  Making it

         15        easier for newer technologies to conform.  But

         16        it is the enemy of precision and that has been

         17        one of the big problems in the past.

         18             >> KING:  Costis.

         19             >> TOREGAS:  I wanted to bring back one

         20        notion that we touched on earlier in the

         21        conversation.  That is the human element in the

         22        systems approach.  And I have been reading and

         23        rereading that sentence, methodology exist to

         24        test voting system software, so we can reliably

         25        demonstrate to operate correctly.  In my own



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                102





          1        experience in complex software deployments have

          2        covered only about 20 percent or so of the

          3        investments made and the anxiety and risk

          4        profiles have to do with hardware and software.

          5        A full 80 percent have to do with the

          6        organizational aspects, the human dimensions.

          7             And I know that it is kind of easy to

          8        sweep under the rug and say we will get to that

          9        later, but let's fix the machine part first.

         10        But I constantly worry about being able to

         11        verify the correct operation system software at

         12        such a high level because all of us understand

         13        and enjoy the discussion and the collegiality

         14        that comes with software engineering and

         15        evaluations and so on.

         16             But ultimately in some precinct, some

         17        polling station, somebody will be pushing a

         18        button or not pushing a button or reading some

         19        kind of an error message or not reading an

         20        error message.  And I guess what I would like

         21        to raise is, is there a role for a prestigious

         22        document like a VVSG to address that human

         23        component or at least put some bounds around it

         24        so we know if we spend so much effort trying to

         25        get to the inth degree or to the tenth decimal



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                103





          1        point of accuracy and precision on the software

          2        reliability, if you will, that we somehow

          3        forget that there is a huge component that has

          4        to do with human dimensions that may perhaps

          5        throw our concerns out the window altogether?

          6        I don't have an answer for that.  But I know I

          7        would love to have some kind of an annex to the

          8        VVSG that addresses the human components to

          9        answer that question about correct operation.

         10             >> KING:  That is an interesting

         11        observation because it certainly gets talked

         12        about.  But whether it gets reduced into the

         13        document in a formal way, that is a very

         14        interesting observation.  Mike.

         15             >> SHAMOS:  So I will propose to answer

         16        the questions that are on the slide.  Do

         17        methodologies exist to test voting system

         18        software so it be reliably demonstrated to

         19        operate correctly?  Sure.  Provided that you do

         20        it during the election.  Parallel testing

         21        achieves that, properly deployed and

         22        administered.  Some more interesting question

         23        is can you tell in advance that the voting

         24        system software is going to operate correctly?

         25        And everything depends on the meaning of the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                104





          1        word reliable.  If reliably means can it be

          2        absolutely demonstrated to be so?  No.  If

          3        reliably means to a sufficient level of

          4        assurance to be able to be used in an election?

          5        I think the answer is yes, although others

          6        differ with that.

          7             What added security benefits are created

          8        by S.I. that are not met by the testing

          9        process?  The answer is there may be profound

         10        benefits, or may be no benefit at all.  It

         11        depends on what the alternative to the software

         12        is.  If the verification and auditing mechanism

         13        is more reliable than the software could have

         14        been, then you certainly have achieved a

         15        benefit.  If the alternative is less reliable

         16        than the software was, then you have achieved

         17        no benefit and in fact you have made things

         18        worse by relying on something that was less

         19        secure than the original.  And as to the issue

         20        of is it easy to determine whether a system is

         21        software independent or not, I am going to

         22        venture to guess that it is touring undecidable

         23        to determine whether a system is software

         24        independent.

         25             There are certainly cases in which you can



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                105





          1        demonstrate that there is software

          2        independence.  For example, in a case where no

          3        piece of software ever touches a document

          4        ballot, it is marked completely by human and

          5        tabulated completely by humans, that is

          6        software independent.  On the other hand where

          7        you have document ballots of the nature of a

          8        VVPAT where there are markings on the ballot

          9        that are created by a piece of software, there

         10        are some things the software might create that

         11        might make the system not software independent

         12        and yet on the other hand there may be ways of

         13        having software write something on a ballot in

         14        such a way that it really is software

         15        independent and I despair it being able to

         16        prove that.  In any case, I still come back to

         17        the point that we have to look at what is the

         18        other thing that we are relying on if it isn't

         19        the software?

         20             >> KING:  Thank you, Mike.  Sarah and then

         21        Daniel.

         22             >> HANCOCK:  Actually it is me.  Because I

         23        don't have a name tag, I am using Sarah's.

         24        Sorry about that.

         25             I just wanted to tag onto what Costis



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                106





          1        brought up about the management side of things

          2        and VAC agrees with that 100 percent.  And we

          3        are developing a set of management guidelines

          4        as a companion document to the VVSG and that

          5        was certainly the intent, they would be used

          6        together.  We are currently in the second year

          7        of a three year initial development phase of

          8        these management guidelines and certainly even

          9        after this initial development stage we will

         10        continue to look at the best practices out

         11        there to continue to increase and make this

         12        document more usable for election officials to

         13        get exactly to the goal that you stated.

         14             >> KING:  Good.  Daniel.

         15             >> CASTRO:  I wanted to address a few

         16        points.  One in terms of the idea of how can

         17        you reliably test if the software dependent

         18        system operates reliably?  It seems within the

         19        VVSG there is already an assumption that there

         20        is a way to do that.  And that is because when

         21        you talk about VVPATs one of the issues is how

         22        do you create an accessible VVPAT?  Within the

         23        VVSG it is that you can use another software

         24        dependent system which would read back the

         25        VVPAT in audio version for the accessible user.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                107





          1             Now, how do you know the best word to

          2        correctly, obviously that is not software

          3        independent and there really is no way to do

          4        that in a software independent way.  If you

          5        want to say the voting is fair and uniform for

          6        all voters, then you have to be making the

          7        statement that, of course, that

          8        software-dependent component of the system is,

          9        in fact, as reliable as the rest of the voting

         10        system, so it seems like within the VVSG, there

         11        is already that hidden assumption that it is

         12        possible to have a software dependent system

         13        that is reliably tested.

         14             I think there is another disconnect in the

         15        VVSG, and that is the idea of having open-ended

         16        vulnerability testing of integrity of the

         17        software.

         18             Even though you're requiring software

         19        independence, there is many reasons to have

         20        open-ended vulnerability testing.  I think that

         21        is a good idea overall.  You want to look at

         22        privacy and availability issues and other

         23        issues that may arise.

         24             If you're talking about testing for

         25        integrity, it seems like, why are you doing



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                108





          1        that if you have the software independence

          2        within the system.

          3             And, finally, I want to amplify a point I

          4        think we brought up a few times, and that is,

          5        as we talked about, many times voters are not

          6        verifying the VVPAT.

          7             So by the definition of software

          8        independence, which is the voting system shall

          9        be software independent, that is, an undetected

         10        error of fault in the voting system software

         11        should not be capable of causing undetectable

         12        change in the election results.

         13             So by that definition, if there is an

         14        error or a fault in the voting system software,

         15        and voters are not checking their VVPATs to

         16        make sure it is correct, then that is violating

         17        the idea of software independence.

         18             >> KING:  Thank you, Peter.

         19             Ron?

         20             >> RIVEST:  I wanted to speak to these

         21        questions of the definition of software

         22        independence.  You've raised some good

         23        questions about the limits of the definition

         24        and when it applies and what counts as software

         25        independent.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                109





          1             In part, the question is, what is the most

          2        useful definition of software independence, and

          3        you need to have a precise definition for

          4        testability, of course, too.  So with the two

          5        issues that were raised -- one is the voters

          6        not checking.

          7             So if a voter does not check his printed

          8        record that he's got if it is printed out, say,

          9        or whatever, it remains as software

         10        independent, because it is detectable.

         11             "Detectable" meaning capable of being

         12        detected.  If the voter is not looking at it,

         13        of course, then he is not detecting it, but it

         14        is still detectable.

         15             So it's just the fact that the voters

         16        don't look at -- the output wouldn't violate

         17        the definition of S.I.  So you can't put a

         18        standard in place that requires the voters to

         19        do certain things and so on.  It is only about

         20        the equipment, so you just have to say it

         21        supports the detections of those kinds of

         22        errors.

         23             So S.I. does not require that the voters

         24        look at the ballots, and of course, it can't.

         25        You can't pass a system or not depending on



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                110





          1        what voters are going to do.

          2             And with respect to voters with

          3        disabilities, you get into a very

          4        interesting -- if you have a blind voter, of

          5        course a blind voter can't look at their

          6        ballot.  And so then the question is:  Is this

          7        going to be SI or not?

          8             The intent is that a system should not be

          9        able to tell if a voter who is voting is blind

         10        or not.  So if a voter is using an interface

         11        that is designed for blind voters, but if a

         12        poll worker goes and votes as if he were a

         13        blind voter and sees the printout himself, then

         14        the error is detectable, and that would qualify

         15        as S.I.

         16             If the voting system knows for sure

         17        whether the voter is blind or not, then, in

         18        fact, it may not be S.I. because the voting

         19        system could change the votes of only the blind

         20        voters.  But if the voting system can't tell if

         21        the voter is blind or not, then you can have a

         22        system, which is S.I.

         23             We're getting into the fine points of the

         24        definitions here.  They are interesting

         25        questions.  But the question basically:  Is



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                111





          1        there evidence produced by the system that

          2        would allow an attempt to cheat, attempt to

          3        defraud somebody, or disenfranchise somebody?

          4             The example that Michael had:  Are those

          5        going to be detectable in principle, and

          6        whether they're actually detected or not is a

          7        separate question.

          8             >> KING:  Peter and Juan.

          9             >> RYAN:  First of all, I wanted to agree

         10        with Mike that, yes, clearly, you have to look

         11        at the rest of the system to see whether

         12        dependency is thrown if you move it away from

         13        the software.  And people are doing that.

         14             We are looking -- taking this sort of

         15        system, a wider view of the system.  So that is

         16        something which is taken on board and people

         17        are working on that.

         18             One other comment I wanted to make about

         19        this one about the assumption or having to

         20        depend on voters performing various checks.

         21        That is one thing that people recognize that

         22        could be an issue, and so we have looked at

         23        other alternatives, so you can supplement, for

         24        example, voter checks by other kind of checks.

         25             So one of the ideas that has been



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                112





          1        suggested is to have sort of a verified

          2        encrypted paper trail, a sort of additional

          3        copies of these protected ballots that is kept,

          4        say, locally, so you can have independent

          5        auditing authorities also making checks at the

          6        cross-points between that and, for example,

          7        what is published on the web bulletin board

          8        that goes into the tabulation, or having voter

          9        assistance organizations, they -- who can vote,

         10        for example, pass their ballot, their

         11        receipt -- protected receipt to their local

         12        representative who could do the check for them.

         13             So there are ways you can supplement the

         14        voter, the dependence on the voters by other

         15        mechanisms if you're concerned about that.

         16             So I think that is my point, yes, and

         17        there are serious issues, but people are taking

         18        those on board.

         19             >> KING:  Juan?

         20             >> GILBERT:  I think there is definitely

         21        other ways to do this.  One thing would be a

         22        multimodal approach.  So getting at the idea

         23        that you have one system or one interface that

         24        everyone votes on independent of ability, in

         25        that case, what a sighted person does would be



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                113





          1        similar or the same as one who is blind, if you

          2        have an interface where you can accommodate

          3        multiple modalities on the same system.

          4             Now, the system can't treat one group any

          5        different because you don't know who is going

          6        to vote how.  So a sighted person could vote as

          7        a blind person.

          8             And these kinds of approaches, I think,

          9        that gets to something, I think Daniel was

         10        bringing up, which is you actually do have a

         11        sense of being able to test and get a

         12        verification that is equal across for everyone.

         13             That should be our goal, and we are not

         14        here to talk about usability.  I understand you

         15        have a different panel to talk about that.

         16             But I think it is kind of -- the two go

         17        together from a usable security perspective.  I

         18        mean, the ultimate secure system is one that no

         19        one can use.

         20             I can secure it in such a way that no one

         21        can use it, I can guarantee it is secure, but

         22        you lose usability, and I can make it so freely

         23        usable and accessible, that it has no security.

         24             So I think, in a sense, those two go

         25        hand-in-hand, and those two have to be



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                114





          1        considered simultaneously.  Otherwise, you're

          2        going to shorthand one or the other.

          3             >> KING:  Thank you.

          4             I have a question for Chris and Sarah, and

          5        it is going back to the first one up there.

          6             If you could consider from either a state

          7        jurisdiction or a county jurisdiction

          8        perspective that ties into a comment that Mike

          9        made earlier about mitigation of risk occurring

         10        at different levels.

         11             So the question that I have is:  If, in

         12        the final version of the VVSG, whatever

         13        software testing methodologies are implemented,

         14        do you see -- among your peers or among your

         15        own colleagues in your jurisdiction, do you see

         16        the development of methodologies at the state

         17        or at the local level to perform testing of

         18        systems that may pick up any residual risks

         19        that are in the systems once they're passed

         20        through the federal certification?

         21             >> THOMAS:  Well, there has been

         22        discussions of doing post-election auditing, of

         23        course, as one way, and the issue of whether

         24        the voter verifies their ballot or not, I mean,

         25        really all of this plays out in a political



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                115





          1        world.

          2             And the political world generally knows

          3        when an election -- when election results don't

          4        conform to a norm that is expected, and that's

          5        when I see the VVPATs coming into real play, is

          6        at the recount.

          7             That is where you're going to be comparing

          8        what the machine says -- the software says

          9        compared to what all of the little pieces of

         10        paper said, as difficult and onerous as that

         11        may be.  So that is an element.

         12             Now, state certification of systems after

         13        the federal system is kind of all over the map.

         14        There is not -- I don't see a lot coming out of

         15        states that I am aware of.  You see what

         16        California has been up to, obviously.

         17             But that is doing much more than once the

         18        system passes, other than to see that it

         19        conforms to the actual state statutes and that

         20        it conforms whatever may be unique or not

         21        tested at the federal level.

         22             Obviously, there is the accuracy tests

         23        that are done.  There is pre- and post-accuracy

         24        tests, and then there is auditing.  And that's

         25        sort of -- the auditing is the next frontier.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                116





          1             Michigan has got one of these pew grants

          2        to work with, Maryland and, I believe, Utah, to

          3        come up with some documentation on

          4        post-election auditing.  But beyond that, I

          5        don't see a lot coming out of the states.

          6             >> KING:  Thank you.  Sarah?

          7             >> JOHNSON:  I agree with my counterpart.

          8        I don't see a lot coming out of states, not

          9        because there isn't a desire or recognition of

         10        how important this issue is.  Not at all for

         11        that reason.  The reason being in most state

         12        agencies that do elections and state boards, in

         13        my case that do certification, the expertise

         14        isn't there.  We don't have the expertise.  We

         15        don't necessarily have access to the scientists

         16        and to all the expertise.

         17             And I am sure Alec can talk about the

         18        money -- the project that it took for Florida

         19        to do their test.  California spent over a

         20        million dollars, etcetera, etcetera.  You have

         21        got the timing issue and the money issues that

         22        all do factor into it.

         23             I don't see a movement that states are

         24        going to start doing more testing or

         25        reinventing the wheel.  I think there is



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                117





          1        already a reliance on the federal testing

          2        system.  We have all learned a lot about what

          3        that testing system was or isn't, so to speak,

          4        but I just don't see the move because of the

          5        expertise and the money.  And we rely on our

          6        state legislators who view or don't view this

          7        as an important issue to do beyond what the

          8        federal does.

          9             >> KING:  I have a follow-up question,

         10        Ron, and then I will turn to you.  John, in the

         11        discussions at NIST, is what Sarah talked about

         12        factored into the scope decision on the VVSG,

         13        the fact that if many states are simply

         14        accepting federal certification as the defacto

         15        state certification that that places an

         16        additional level of expectation on the federal

         17        VVSG?

         18             >> WACK:  That -- I don't know how best to

         19        answer that.

         20             >> THOMAS:  The answer is yes.

         21             >> WACK:  To a certain extent, I would say

         22        that the big area where I heard discussions was

         23        really more in additional testing that states

         24        are doing on top of federal certification

         25        testing.  And that some states were actually



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                118





          1        doing some fairly expensive expert security

          2        reviews and additional banging away and in

          3        finding some real problems, finding some fairly

          4        significant problems and issues that would lead

          5        them to wonder why the systems passed the

          6        certification test to begin with.

          7             And that this was expensive and if the

          8        VVSG could essentially set up a system so that

          9        the conformance testing to the VVSG would in

         10        itself sufficiently rigorous, states may not

         11        have to do additional testing.  Or may not have

         12        to do additional testing to the extent that

         13        some states were doing.  So that it might be

         14        ultimately less expensive to have a more

         15        rigorous VVSG and a more rigorous performance

         16        testing process that, ultimately, you know,

         17        those savings would be passed onto states.  I

         18        don't know if that really answers your

         19        question.

         20             >> KING:  John, does rigorous there imply

         21        depth, more breadth or both?

         22             >> WACK:  Well, I would say that both.

         23        Breadth in terms of volume testing.  Previous

         24        versions of the standards allowed bypassing

         25        certain parts of the system in testing.  You



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                119





          1        could hook a test harness up to the system and

          2        bypass the user interface and not have to test

          3        that to the same extent.

          4             Some of the accuracy tests were specific

          5        accuracy tests, and the new approach is really

          6        to in essence, like I said earlier, conduct the

          7        mock election and pretty much have live test

          8        subjects banging away on systems for a period

          9        of time, starting with the user interface and

         10        ending up with making sure that you are getting

         11        usable records out of the voting system.  So

         12        that is the breadth.

         13             The depth, to a certain extent is handled

         14        in the usability performance testing for

         15        accuracy.  And open-ended vulnerability

         16        testing, I have sometimes wished that it had

         17        been called an expert security review because I

         18        think the open-ended part of it leads people to

         19        believe that it is, in a very expensive

         20        open-ended process that will never end.  But it

         21        is an expert security review that in essence

         22        allows people to dive down into the system if

         23        they need to find out if there are any issues.

         24        So I guess my answer is, I think to the extent

         25        that people could, they try to make it wide and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                120





          1        deep.

          2             >> KING:  Okay.  Thank you.  Ron.

          3             >> RIVEST:  I think John actually made

          4        most of the points I am about to make.  But I

          5        just wanted to emphasize them a bit.  I think

          6        one of nice things about the new VVSG is the

          7        volume testing that is going on.  It really is

          8        running a mock election as John said, and I

          9        think that is a major help in trying to weed

         10        out systems that are unreliable when you have

         11        got frequent printer jams or other problems.  I

         12        think that just running a heavy duty mock

         13        election with real equipment with real voters a

         14        major improvement.

         15             And I think it is modeled, back to the

         16        question of tension between the state and

         17        federal -- or not tensions but cooperation

         18        between those two, it really is modeled after

         19        the California volume testing.  So the states

         20        are leading the way in some of these things I

         21        guess is where the federal government is,

         22        adopted and picking it up and saying volume

         23        testing is a good thing to be doing here.

         24             >> TOREGAS:  If can I ask a clarifying

         25        question.  Have the legal people opined about



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                121





          1        liability, shared liability, if state or

          2        municipal government, county government assumes

          3        that a test that was done at the federal level

          4        would ferret out all of these inconsistencies

          5        and in fact was proven not to?  What happens?

          6        I am sure there is an easy answer to that but.

          7             >> KING:  I will defer that to counselor

          8        Shamos.

          9             >> SHAMOS:  There is not an easy answer.

         10        What happens is we had an example of this in

         11        Pennsylvania where a system that had been

         12        certified was suddenly decertified when certain

         13        flaws were pointed out by citizens.  A

         14        re-examination was conducted and was

         15        decertified.  That was easy to do.  Then the

         16        counties that had purchased the system showed

         17        up at the door of the secretary of the

         18        commonwealth and said, we need money to buy

         19        another system that you said was certifiable in

         20        place of the one that you originally said was

         21        certifiable.

         22             And in that particular case the number of

         23        counties in which the system was used was small

         24        enough that the secretary was willing to pay

         25        the freight.  But if half the counties in the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                122





          1        state had used that system, we would still be

          2        in litigation over it.

          3             It is absolutely unclear whether there

          4        is -- could be any liability on the secretary

          5        of state for carrying out secretary of state's

          6        duties.  Could there be any liability on for

          7        example the consultant to the secretary of

          8        state who actually performed the examination?

          9        I shudder to think.

         10             >> KING:  Alec.

         11             >> YASINSAC:  I just wanted to follow-up.

         12        I think there is a bit of a misconception about

         13        the OEVT and what has happened in the states.

         14        To my knowledge, no state does OEVT testing as

         15        part of their certification process.

         16             The TTVR in California was all systems

         17        that had been in use and the question was

         18        decertification process, not the certification

         19        process where the plan was laid out ahead of

         20        time.  I think this is a critical point because

         21        in the future it was -- it is actually fairly

         22        easy to find flaws when nobody knows you are

         23        going to be looking for flaws.

         24             Once this notion is applied and once if it

         25        goes the way it would -- would be most



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                123





          1        beneficial in my belief, then the process would

          2        mature so that the OEVTs are guaranteed to find

          3        things.  I mean, it would be hard to put

          4        together a team almost with the systems that

          5        have been out there.  They won't find faults in

          6        the process, the TTVR, the Florida studies,

          7        there have been things that have found flaws

          8        because the systems weren't engineered to be

          9        able to defend against that type of analysis.

         10             It was done after the fact and so it was

         11        very easy to find those faults.  So the next

         12        thing we have to look at is what happens when

         13        you do an OEVT and don't find faults?  Then

         14        that is the big question there.  But it is not

         15        exactly an implementation on what the states

         16        have done.  It is kind of a reversal that says

         17        we need to do it up front and hopefully that

         18        will impact the process.

         19             >> KING:  Ron.

         20             >> RIVEST:  I was referring to the volume

         21        testing California does, not the top to top

         22        bottom review.  You are absolutely right that

         23        that was post facto.  The volume testing I

         24        believe in California is done before

         25        certification in California.  But I may be



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                124





          1        mistaken on that.

          2             >> YASINSAC:  I am not sure.

          3             >> TOREGAS:  One more question.  One more

          4        quick question.  Again, to signify my

          5        ignorance.  Where is the industry in all of

          6        this in terms of demonstration of reliability

          7        and so on?

          8             I presume that there is a certain finite

          9        number of companies that are involved in this

         10        business, and I, again, presume that the VVSG

         11        is ridden with an eye towards the industry

         12        itself.  Can somebody kind give me an idea of

         13        where the industry is in terms of their current

         14        ability to demonstrate correct operation and

         15        their ability to operate correctly under the

         16        proposed VVSG?

         17             >> KING:  Well, there are no vendors

         18        present at the table here.

         19             I don't know, Brian, is that something you

         20        want to comment on in term of the vendors'

         21        roles in the VVSG?

         22             >> HANCOCK:  Well, the only comment that I

         23        will make is the structure of the TGDC was laid

         24        out in the Help America Vote Act, as we all

         25        know.  And as TGDC is constructed, there is no



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                125





          1        vendor involvement at the table, although

          2        certainly in 2005, we received numerous

          3        comments from the vendor community, and we

          4        certainly expect to receive at least as many,

          5        if not more comments this time around.

          6             >> KING:  Yes, I think cost is something

          7        that you mentioned earlier, and we've had some

          8        side conversations about, and that is:  Who are

          9        the stakeholders in this process?

         10             And obviously the vendors are a

         11        stakeholder.  But then the follow-on question

         12        is:  What is their role?  What is their

         13        responsibilities?  What is their

         14        accountability?  What is the state of the

         15        practice?

         16             Those are all related to that essential

         17        question of who are the stakeholders.

         18             >> TOREGUS:  I guess the reason I ask it

         19        is not because I am speaking for the vendor

         20        community.  I am not, because I don't belong to

         21        the vendor community.

         22             But if I had a 25-year-old son -- I have a

         23        20-year-old son, so I am getting there -- if I

         24        had a 25-year-old son.  And he said, Dad,

         25        should I go into this business?  Is this a good



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                126





          1        business for me?  Is this an entrepreneurial

          2        business to get involved in?

          3             And having heard this discussion and

          4        having read some of the materials, I would say,

          5        boy, that is a real rough area to promote to my

          6        25-year-old son.

          7             If I am the secretary for economic

          8        development in the state or at the federal

          9        level, and I want to promote and to strengthen

         10        industry, it's part of the harsh balancing act

         11        between standard setting organization that

         12        tries to develop kind of aggressive and correct

         13        things, but at the same time, be able to find

         14        the actual product in the marketplace.

         15             And unless the states and localities of

         16        the federal government is prepared to

         17        manufacture these things, we also have to give

         18        ear to that.  I hear your discussion that,

         19        historically speaking, they have not been given

         20        a place on the table.

         21             But ultimately, there has to be some kind

         22        of an intervention, some kind of listening ear,

         23        and I would suggest that would be useful again,

         24        part of the expanding communication for this

         25        discussion.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                127





          1             >> KING:  And well said.

          2             John?

          3             >> WACK:  I had one other comment, and

          4        that is, the question up there, "Do

          5        methodologies exist to test voting system

          6        software so it can be reliably demonstrated to

          7        operate correctly," was -- that question was

          8        pondered a lot.

          9             Actually, one of the people working with

         10        NIST in the TGDC went down the route of

         11        essentially wanting to develop systems in such

         12        a manner that they can be proved through some

         13        sort of a fault analysis to be correct, and

         14        they, you know, possibly may not require any

         15        sort of independent record.  This analysis

         16        would be sufficient.

         17             And the little I know about this area -- I

         18        know it is pretty foolhardy to talk about

         19        something you don't really know a whole lot

         20        about in public.  I think peter Ryan knows

         21        probably more about what I have to say than I

         22        certainly do.

         23             But if you are going to basically develop

         24        software that you can test to reliably

         25        demonstrate to be correct, you have to develop



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                128





          1        it in a way, in a very specific way, such that

          2        it can actually be tested.  So you probably

          3        have to use very formal methods, which the DOD

          4        and I am sure the airline manufacturers use, to

          5        develop software in very specific ways in which

          6        it can be more easily tested.

          7             And ultimately -- you know, the E.A.C.

          8        ultimately has to end up with a document that

          9        testers can very clearly read, and it can't be

         10        fuzzy testing.  It has to be fairly specific

         11        testing.  It has to know what passes and what

         12        doesn't.

         13             So the reason I say that basically is we

         14        could have gone down that approach with the

         15        VVSG, possibly, and really upped the way in

         16        which software was developed and specified it

         17        and possibly could have made it easier and more

         18        accurately tested.

         19             However, that would have come at great

         20        expense, quite a bit of an expense.  It would

         21        have taken much longer to develop the VVSG, and

         22        it may not have been something that the vendors

         23        would have wanted to accept, because they would

         24        have had to change their software development

         25        practices in a big way, and it would be much



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                129





          1        more expensive for a vendor to get into the

          2        business than currently.

          3             So, basically I bring this up just to say

          4        that, yes, I believe there are methodologies

          5        out there to test software to reliably

          6        demonstrate whether it is correct, but it means

          7        that it has to be developed in specific ways to

          8        make that possible, and that will be very

          9        expensive.

         10             >> KING:  John, thank you.

         11             I wanted to make a comment that ties

         12        together John and Alec's observation about the

         13        possible challenges of applying open-ended

         14        testing to products that were not designed to

         15        sustain that kind of scrutiny.

         16             And John said we need to be looking at how

         17        the VVSG and its implementation impacts vendors

         18        coming into the market space.  And I think for

         19        those of us who rely upon these vendors, I

         20        think there is an additional issue of having

         21        those vendors stay in the market space.

         22             And one of the concerns that is expressed

         23        often by election officials is:  What are the

         24        contingency plans if vendors decide that they

         25        are going to move their resources into other



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                130





          1        markets, that they may be more competitive in?

          2             So I do recognize that the folks that are

          3        crafting the draft of the VVSG do think about

          4        the vendors' role and often think about it in a

          5        way of just trying to understand the vendors'

          6        role, and sometimes the vendor doesn't fully

          7        understand the vendors' role, it seems, so a

          8        very complicated problem.

          9             I would like to move on now, if we could,

         10        to the next question on the list, question

         11        number 4:  "What are the relative merits of the

         12        various types of direct, that is by the voter,

         13        and indirect by automated system independent

         14        verification techniques?"

         15             And we didn't have a volunteer for there,

         16        so I will make a few introductory comments, and

         17        I'll turn it over to the other members of the

         18        panel.

         19             When I thought about the direct

         20        verification, there is a lot of benefit to

         21        having the voter directly involved in the

         22        verification of the ballot.

         23             One is that the verification techniques

         24        can be very intuitive, may require a small

         25        amount of training and/or easily understood by



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                131





          1        the voters, that is, what they're verifying.

          2             The second is that they can be realtime,

          3        and in transaction processing systems, we often

          4        talk about the inherent challenges of

          5        pre-editing and post-editing.

          6             And that is, when you can pre-edit data,

          7        that is, get the right data, correct data into

          8        the system, it optimizes the processing and

          9        minimizes the need for exception handling.

         10             On the other hand, if the edit is kind of

         11        post-processing, that involves not only methods

         12        of handling the air conditions of the

         13        anomalies, but then you have to have a method

         14        of verifying that the verification has, in

         15        fact, occurred and any changes have been done

         16        properly.

         17             So some questions that I wrote down is:

         18        Can correct verification approaches be enhanced

         19        by better ballot design?  And again, I think in

         20        part that is a usability question and an

         21        accessibility question.

         22             But, I think, also, when we ask voters to

         23        verify what they have done -- we have heard on

         24        this panel and other discussions -- that voters

         25        often do not verify.  They don't consult the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                132





          1        VVPAT, so is that possibly -- instead of

          2        sometimes it is assumed to be an issue of voter

          3        disinterest, could that be issues related to

          4        ballot design?  Could it be issues related to

          5        navigational schemes within the system, related

          6        to voter education initiatives?

          7             One of the things that we -- we often talk

          8        about, is one of the holy grails in voter

          9        verification, is the notion of the secret

         10        ballot.

         11             And my experience is that there are a lot

         12        of different ways that we define the secret

         13        ballot.  Some of it is a technological

         14        definition.  Some of it is certainly a legal

         15        statutory definition.

         16             But do we have consensus on what we mean

         17        by secrecy of the ballot?  And even when we

         18        talk about the ballot, verifying --

         19        verification of the voter's intent,

         20        verification of the voter's choices,

         21        verification of the voter's tabulated choices,

         22        there is so many different dimensions to that

         23        that again, a theme that I have heard here

         24        today is that perhaps a lexicon in models would

         25        be helpful in reaching consensus.  The indirect



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                133





          1        methods of voter verification obviously

          2        introduce a different system, whether that

          3        system is manual or automated or a hybrid of

          4        that.  And when we introduce an additional

          5        system then we have all of the uncertainty that

          6        a new system creates not only its inherent

          7        functionality but its interfaces to the

          8        existing system become source of risk.

          9             And my final question that I have to help

         10        the panel kind of form their responses is going

         11        back to something that Mike said and I said

         12        earlier and it deals with reasonable assurance.

         13        When we are talking about verification, whether

         14        direct or indirect, what is our target?  Is it

         15        our target that absolute zero uncertainty or

         16        are we moving toward discussions of reasonable

         17        assurance, reasonable test and what might those

         18        be?  So with those comments to kind of help us

         19        form some questions, I will turn it over to the

         20        panel.  Mike.

         21             >> SHAMOS:  I will plead again for some --

         22        a definitional beginning of exactly what we

         23        mean by verification.  So I think what most

         24        voters believe is going on with their

         25        verification is not what is actually going on.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                134





          1        There is a common phrase that is used but

          2        voters want to be sure that their ballot was

          3        counted as cast.  That is the phrase.  Counted

          4        as cast.  There is no system currently that

          5        provides that kind of verification, except the

          6        cryptographic ones.  I am talking about the

          7        commercially available systems do not provide

          8        that.  What they do provide is a verification

          9        that the system correctly captured the voters

         10        intent.  Because if the system is able to

         11        capture it and spit it back out to the voters

         12        so the voter can check and see that all the

         13        selections are the ones the voter wanted to

         14        make, that indeed demonstrates to the voter

         15        that she was heard correctly.

         16             As to what happens downstream, the voter

         17        does not have assurance of that.  In fact, we

         18        want to make sure that the vote is counted

         19        correctly.  We want to make sure that whatever

         20        record is made of the vote is sufficiently

         21        permanent, that it still exists not only at the

         22        end of the election but at the time of any

         23        audit or recount.  And we want to make sure and

         24        this is virtually never spoken by anybody, we

         25        want to make sure that no unauthorized voters



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                135





          1        voted in this election because it doesn't do me

          2        any good if I know that my vote was counted if

          3        ten people vote for every living person, then

          4        it doesn't matter whether your vote was counted

          5        correctly.

          6             And further more, all of this stuff needs

          7        to be able to verified by the voter after the

          8        election is over.  And typically, what the

          9        VVPAT provides is the first form of

         10        verification, which is the system heard me

         11        correctly.  So in answer question four, I am

         12        interested in exactly what kind of verification

         13        are we talking about?  And my preference would

         14        be for end-to-end verification.  But there are

         15        no commercial systems to provide that.

         16             >> KING:  If I can ask just the following

         17        question.  Mike, do you think there are legal

         18        barriers to that end-to-end verification?

         19             >> SHAMOS:  No.

         20             >> KING:  Okay.

         21             >> SHAMOS:  I think what you are saying

         22        is, if the voter can satisfy herself that her

         23        vote was counted then it seems to stand to

         24        reason that she ought to be able to prove that

         25        to her neighbor that her vote was this and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                136





          1        counted that way.

          2             But there are cryptographic systems that

          3        permit solo verification without the ability to

          4        prove to someone else how the voter voted.  So

          5        was that the concern?  The secrecy?  Also I

          6        agree we need a definition of secrecy.  Secrecy

          7        seems to mean two things.  To some people it

          8        means that no one else can find out how the

          9        voter voted without the voter's permission.

         10        And to me, what secrecy means is not only can

         11        no one else find out, but the voter cannot

         12        demonstrate to another person how she voted.

         13        And I think those are often intermixed, and we

         14        ought to keep them straight, secrecy one and

         15        secrecy two or something like that.

         16             >> KING:  Thank you.  Ron and then Peter.

         17             >> RIVEST:  Thanks.  I think Mike started

         18        off on a good direction here.  It is often

         19        helpful to have clear definitions as to what is

         20        being verified and what we are talking about

         21        and I agree with everything he said on those

         22        points.  Just to try to clarify some of the

         23        potential distinctions one could make, when I

         24        talk about things that might be verified in

         25        voting I tend to make a following three-way set



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                137





          1        of steps that need to be verified which are

          2        pretty much -- first is cast is intended.  Is

          3        the vote captured correct?  Did the vote get

          4        correctly captured.  That can be direct or

          5        indirect as we have talked about.  So cast is

          6        intended.

          7             And the second step which is often skipped

          8        as collected as cast.  So usually there is a

          9        process of collecting the records before they

         10        are counted so are they properly collected?  Is

         11        there -- and some of the end-to-end systems

         12        have a formalization of that process where in

         13        fact all of the ballots that were cast

         14        encrypted form to the website so you can

         15        actually see the collection and you can verify

         16        that a particular ballot is present in that

         17        collection.  So verifying that they are

         18        collected as cast is a verifiable step in some

         19        of these systems.

         20             And then finally, you have the step

         21        counted as collected.  So you want to verify

         22        that the tabulation is correct.  That is

         23        interesting if the ballots are encrypted of

         24        course, but there are ways of getting around

         25        that.  So count cast as intended, collected as



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                138





          1        cast, counted as collected I think is an

          2        interesting and useful set of distinctions to

          3        make.  The issue about no dead voters is

          4        probably the one of collected as cast.  I guess

          5        you have got things in the collection which

          6        correspond to things that weren't cast and so

          7        have to go there.  I think the best way to do

          8        that is to post a list of the voters who

          9        allegedly voted and have other citizens be able

         10        to check that and say, you know, this person

         11        didn't vote because they are dead or something.

         12        But that is a hard one.

         13             >> KING:  Okay.  Thank you.  Peter.

         14             >> RYAN:  Well, I agree very much with

         15        what Mike said and what Ron just said.  I want

         16        to follow-up your comment about the secrecy.

         17        That is a very good point and people have

         18        recognized that and there are at least three

         19        different flavors which correspond I think very

         20        precisely with what you just said.  Privacy in

         21        the naive sense, a passive adversary, and there

         22        is an even more sophisticated one called

         23        cohesion resistance so that has been taken up

         24        and people have come up with precise

         25        definitions of these different



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                139





          1        characterizations of security.  So that is a

          2        good point.

          3             Yeah, I would just like to echo what Ron

          4        said and the way I would like to describe some

          5        of those cryptographic systems as they are in

          6        the sense three stages in which the counting,

          7        collecting, and counting of votes that can go

          8        wrong.  First in the encryption or encoding

          9        step that things can go wrong and then sort of

         10        transmission into the tabulation phase, and

         11        then the actual tabulation and encryption

         12        phase.  So you have to look at all three of

         13        those and make sure that there are ways to do

         14        checks and balances to make sure that any

         15        malfunctions at any of those stages can be

         16        detected.  So that is in essence what is going

         17        on at a very high level.

         18             >> KING:  And Mike, if I am not mistaken,

         19        there are jurisdictions where the vote of the

         20        dead can be counted depending on when they cast

         21        it in early voting.  So you may need to expand

         22        that explanation.  I wanted to pose another

         23        question to the panel that is related to this.

         24        And I think it is a good question because it

         25        doesn't have an obvious answer.  And that is,



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                140





          1        can mandates produce technology?  And this has

          2        come up on a couple of different discussions.

          3             >> SHAMOS:  Mandates plus money.

          4             >> KING:  And if it can produce

          5        technologies, are they appropriate

          6        technologies?  And by that I am always looking

          7        for the application of the law of unintended

          8        consequence.  So as we look at the VVSG and

          9        perhaps other things that kind of swirl around

         10        federal issues in election technology, be

         11        interested in the panel's thoughts about the

         12        efficacy of mandating technology and evolution

         13        through statute or through incentives plus

         14        statute and whether there are any potential

         15        unintended consequence of that.  And I can't

         16        tell if these are up for this question or

         17        for -- okay.  Mike, go ahead.

         18             >> SHAMOS:  Unintended consequence.  So

         19        when VRE machines first came into use in the

         20        early 1980s, almost all states that allowed

         21        them had a requirement that they had to

         22        maintain a paper record of vote, individual

         23        cast vote records.  And the vendors recognized

         24        at that time that it was impossible to maintain

         25        that record in sequential form where it -- or



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                141





          1        would be possible to tie a particular cast vote

          2        record back to a particular voter, so various

          3        randomization techniques were used, and they

          4        were used consistently in all D.R.E. machines

          5        until after the year 2000, when the call came

          6        for voter verified paper records and the

          7        vendors saw that they had a sales opportunity,

          8        they threw away absolutely everything they had

          9        always understood about the need for

         10        randomization and immediately came out with

         11        inexpensive, sequential paper rolls, which they

         12        knew they couldn't use, but nonetheless, it was

         13        a solution to a mandate, it was a solution to a

         14        problem, and look what has happened.  And now,

         15        they're all over the place.

         16             And so the unintended consequence is that

         17        some people have woken up and said, wait a

         18        minute, this violates voter privacy, we can't

         19        ever actually let anybody look at that

         20        sequential paper trail, or it will violate

         21        voter privacy.

         22             So it is stored away in a safe, and nobody

         23        ever gets to use it.  They don't use it for

         24        audits.  They don't use it for anything.  I

         25        think that it is a highly unintended



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                142





          1        consequence, but it is because the vendors were

          2        forced to do something very quickly to respond

          3        to a perceived market need.

          4             And everybody threw away the cautions of

          5        the past and allowed them to do it, and then

          6        actually bought the systems and then actually

          7        used the systems.

          8             >> KING:  All right.  Thank you, Mike.

          9             Any other discussion on this question?

         10        Yes?

         11             >> THOMAS:  I just agree with Mike.  I

         12        think that as this political process moves down

         13        the road in terms of defining what these

         14        systems do is, watch out what you're going to

         15        get, and I think also watch out for undermining

         16        further any confidence that people are going to

         17        have.

         18             The more this goes down the road, the more

         19        discussions that are out there and kind of all

         20        of the boogie man stuff that goes on with it.

         21        It is doing an incredible job of undermining

         22        things.

         23             >> KING:  Costis?

         24             >> TOREGUS:  A reaction you provoked with

         25        your question, and then something else.  In



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                143





          1        other sectors when we think about new

          2        technology, there is usually an opportunity to

          3        do what the social engineers would call

          4        "business process re-engineering."

          5             In other words, look at the process as it

          6        existed before the new technology comes on

          7        board, and then have a chance to change the

          8        process because of what the technology enables

          9        you to do.

         10             It appears to me that in this particular

         11        sector, election technology sector, we're

         12        trying to kind of have our cake and eat it,

         13        too.  We think about innovation, maybe some

         14        changes, but it is usually at the fringes.

         15             I would be hard-pressed to remember a

         16        paper I read or a product that I saw in the

         17        marketplace that makes fundamental changes to

         18        the process itself.

         19             And of course, we could quickly say, well,

         20        the process is sacrosanct.  This isn't the law.

         21        This isn't how we do it.  And yet, in many

         22        other areas of government, we have found ways

         23        to modify the process and take advantage of the

         24        existing technology.

         25             For me, at least, I find the technology of



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                144





          1        elections has not yet matured enough, perhaps,

          2        to be allowed to enter this realm of thinking

          3        of business process re-engineering, and perhaps

          4        that is something to be considered in the fifth

          5        question, we talk about the innovation class.

          6             The other comment that I have is with

          7        these kind of evaluations and verification and

          8        so on.  After a certain point, we have to also

          9        begin to think about how the voter thinks.

         10             And I have to be honest with you as a

         11        voter myself, I would much rather go to an

         12        election place, cast my vote, and have the

         13        confidence and the expectation that the system

         14        that receives my vote, the public

         15        administration system behind that process is

         16        strong enough, transparent enough, and

         17        professional enough to take care of everything

         18        else.  It is a hope.

         19             But in a sense, here we are, trying to

         20        manufacture the technology component to do

         21        something that perhaps we ought to be looking

         22        at strengthening our public administration

         23        system that could then run circles around what

         24        a simple machine can or cannot do.  That is a

         25        bigger, a higher level of expectation that



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                145





          1        perhaps some other group needs to be thinking

          2        about.

          3             >> KING:  Yeah, I am going to take this

          4        opportunity to plug the management guidelines

          5        effort of the E.A.C., which I think that's a

          6        fairly good description of what the goals of

          7        that process is.

          8             Dan?

          9             >> CASTRO:  I just wanted to add on to

         10        what Mike said earlier about can mandates

         11        produce technology, and, yes, but the funding

         12        doesn't matter.

         13             And I think it is interesting, because

         14        with voting technology, the way we're setting

         15        the standards here is very different than what

         16        you see in a lot of other industries.

         17             For example, in the auto industry, when

         18        they're setting fuel efficiency standards, that

         19        is set for cars that will be developed in the

         20        future.  It is not imposing any kind of

         21        standards on the existing technology.  I think

         22        it is very important when you talk about

         23        setting standards that don't have any -- any

         24        funding tied to it.

         25             I also think it is important, before you



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                146





          1        set those standards -- maybe there is a need

          2        for it -- but when you decide that there is a

          3        need, you have to be deciding that based on a

          4        clear risk analysis, which is what we

          5        originally talked about.  You have to say this

          6        standard is this much measurably better than

          7        what we had before to justify that kind of

          8        cost.

          9             >> KING:  That is a good point.

         10             When I was formulating the question on --

         11        you know, can you guide the evolution of

         12        technology with mandates, I did think about the

         13        fuel efficiency and the '73 Pinto and '74 Vega,

         14        and I do have concerns about how effective it

         15        is to mandate the development of technologies.

         16             I think what I would like to do at this

         17        point is, I am looking at the -- at the body

         18        language of the panelists, and I think it is

         19        time for another break.

         20             And I would like this 15-minute break to

         21        be a little shorter than the last 15-minute

         22        break, because we do -- we still have another

         23        question, and then we have some summarization

         24        to go through, but I do think people need to

         25        stand and stretch, particularly up here up



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                147





          1        front.

          2             So if we could, let's take a hard 15

          3        and --

          4             >> AUDIENCE MEMBER:  What time do we have

          5        to be back?

          6             >> KING:  Good point.  I have 4:25.  Let's

          7        be back at 4:40.

          8             (Recess from 4:25 to 4:40 p.m.)

          9             >> KING:  If we can take our seats please.

         10        Before we start in on the fifth and final

         11        question, I wanted to kind of recap the agenda

         12        and where we need to go and what we want to try

         13        to finish up in the next hour or so.  After we

         14        have addressed question five, we have left

         15        enough time in the schedule to come back and

         16        really maybe tease apart some of the points we

         17        looked at earlier that perhaps subsequent

         18        discussions of questions illuminated something

         19        that we want to go back through and review.

         20             And then before we leave today, we have

         21        got an opportunity for every panelist to make a

         22        closing statement.  And my intent there was

         23        usually when I am giving a lecture, there is

         24        two or three things that I want to make sure

         25        that the students walk away from and often I



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                148





          1        refer to those as just the take away.  So there

          2        will be an opportunity at the very end for each

          3        of to us pass onto our colleagues on the panel,

          4        pass along to the E.A.C. Standards Board and to

          5        pass on to the E.A.C. commissioner the things

          6        that we really would hope that will be

          7        persistent from this meeting and merit further

          8        attention and perhaps further discussion at

          9        subsequent meetings.

         10             So I am going to -- Peter, I am hoping

         11        that your biological clock is revving up around

         12        now.  Mine is starting to -- well, that you are

         13        on UK time.  No.  I am sorry, it is very late,

         14        isn't it?  But Peter has volunteered, and we do

         15        have question five up there.  He has

         16        volunteered to lead us into opening discussion

         17        of that question.  So if you would.  Oh, you

         18        thought you had?  Then I accept your

         19        withdrawal.  I am sorry.  We had discussed this

         20        earlier.  So I will do my best to get things

         21        started but jump in and help me.

         22             The question is how can innovative systems

         23        be evaluated for purposes of certification?

         24        And some follow-up questions about how do other

         25        industries deal with testing and certification



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                149





          1        of innovative products?  And this is I think

          2        directed towards the concept of an innovation

          3        class as an alternative way of introducing

          4        systems into the VVSG certification process.

          5        Do we create unintended back doors for the

          6        certification process and again looking ahead

          7        at potential unintended consequence?  And then

          8        can a set of limited standards be created in

          9        order to make the path toward certification of

         10        innovative systems more clear?

         11             And I think in looking back at the 2005

         12        VVSG, I had an opportunity to work with the

         13        E.A.C. on the collecting and cataloging of

         14        comments.  And I can remember distinctly there

         15        was a day about 45 days out from the end of

         16        comments where I was asked, you know, is there

         17        much activity?  And the answer was no, it was

         18        just some people are responding.  And then as

         19        people really thought through the implications

         20        of what was in that document, it accelerated.

         21        And then as a function of that acceleration,

         22        other people saw the comments that were being

         23        posted and that inspired additional lines of

         24        thinking.

         25             So I know that the authors and the TGDC



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                150





          1        and the Standards Board are all concerned that

          2        as this document goes forward, that due

          3        diligence has been done on thinking through

          4        unintended consequence.  We have got always the

          5        mandate that we have to live with these

          6        documents for whatever their life is.  And so I

          7        think with that as an introduction about how is

          8        innovation managed perhaps in other industries?

          9        Are there models that can be looked at?  And

         10        does the innovation class proposal create a

         11        potential back door to the certification

         12        process?  And then finally, is there potential

         13        limited set of standards that could accommodate

         14        the innovation class approach?

         15             >> SHAMOS:  I have now forfeited my right

         16        to talk.

         17             >> KING:  I have been waiting all day for

         18        that to happen.  Recognize my -- go ahead.

         19             >> SHAMOS:  So I am not a fan of the

         20        innovation class or the name innovation class.

         21        So in fact I think it is a proof that possibly

         22        too much of the VVSG is design oriented rather

         23        than functional oriented.  Because if it were

         24        truly functionally oriented then anybody could

         25        propose a system for certification, and it



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                151





          1        would be examined to see if it performed the

          2        necessary function.

          3             If one of functions is software

          4        independence, fine.  If one of functions is

          5        that it must provide verifiability at certain

          6        levels in the voting process then fine.  But I

          7        think what is going on is the VVSG has anointed

          8        VVPAT systems as the ones that satisfy the

          9        requirements.  And if you are not a VVPAT, then

         10        it is your obligation to show up hat in hand

         11        and prove to the examiners that your system

         12        indeed satisfies these functions.

         13             And I don't think there is any reason at

         14        all to make that -- to make that distinction.

         15        It is conceivable that somebody could come up

         16        with a system that didn't fit into the VVSG at

         17        all.  Hypothetically, suppose I had a reaping

         18        machine that was able -- I could point it at

         19        your head and determine from that how you

         20        wanted to vote.  Such a system, frightening

         21        though it might be, if it were submitted for

         22        certification I don't believe it could be

         23        tested to the 2007 VVSG which doesn't even

         24        contemplate such a thing.

         25             Okay.  But yet, the systems that we are



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                152





          1        talking about, for example, the prime three

          2        system at Auburn, is perfectly consistent with

          3        the kinds of voting systems that we have seen

          4        before and doesn't need to be sent to the back

          5        of bus in the innovation class.

          6             As far as whether a certification process

          7        for innovative systems is going to be a back

          8        door, I think it is quite the other way around.

          9        I think that the -- that what happens in these

         10        so called innovative systems -- and I am not

         11        sure what you mean by innovative.  I think it

         12        was an effort to sugar coat what was really

         13        going on.  But let's take it at face value and

         14        if these innovative systems were truly

         15        innovative, that is they incorporate new ideas

         16        and new ways of achieving security, new ways of

         17        achieving assistive voting, new ways of

         18        achieving verifiability, then it certainly

         19        isn't going be a back door that will allow

         20        these fundamentally bogus and substandard

         21        systems to somehow get into the elevated class

         22        of the VVPAT.  It will not happen that way at

         23        all.

         24             My belief is that these systems will end

         25        up being better, and it is not a back door.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                153





          1        And in fact, the VVPATs that have already taken

          2        the back door.  And so can a set of limited

          3        standards be created in order to make the path

          4        toward certification more clear?  I don't think

          5        you need a set of limited standards?  I

          6        wouldn't suggest that the standards ought to be

          7        limited.

          8             We have certain functional requirements

          9        that we believe need to be met.  A system that

         10        meets them gets certified and a system that

         11        doesn't, doesn't.  That is my view of it.

         12             >> KING:  Thank you, Pete.

         13             >> RYAN:  I will have a try and make some

         14        comments.  The more I hear, the more I am glad

         15        I didn't try to lead on this.  Because I

         16        realize there are all kinds of hidden agendas

         17        and things which I didn't appreciate at the

         18        outset.

         19             Well, let me speak to the first question,

         20        how do you evaluate these individual systems?

         21        Strikes me that that's profoundly difficult

         22        problem.  I think it goes back to the

         23        discussion we were having this morning about

         24        how do you set up your threat models and so on

         25        and so forth.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                154





          1             So I think I would just suggest -- that

          2        strikes me as a very difficult one.  I don't

          3        know if you are expecting a real answer.  The

          4        only thing that comes to my mind is that

          5        actually is going to require a substantial

          6        amount of research to come up with systematic

          7        ways in developing threat models, which will --

          8        in a sense, it comes back to your question

          9        earlier.

         10             I think the threat models and so forth are

         11        dependent on the systems.  If you come up with

         12        completely new kinds of technologies, you

         13        introduce totally new classes of threat and so

         14        forth.  So I think this is a profoundly

         15        difficult problem, will be my first comment.

         16             Yes, I think I'll leave it for that.  And

         17        hand it over to someone else for a minute.

         18             >> KING:  I'll respond.  I think that

         19        is -- it is a challenge, as we're trying to, if

         20        you will, kind of wrap our minds around the

         21        risk assessment threat assessment models, for

         22        lack of a better term, for conventional systems

         23        moving through the VVSG process, what will be

         24        the threshold of uniqueness in an innovation

         25        system that will require the research of a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                155





          1        different set of models perhaps to validate it.

          2             So from -- from my perspective, as I'm

          3        looking at -- not just the current VVSG but the

          4        past and future ones -- is trying to understand

          5        the overhead that is embedded into the

          6        application of the VVSG and how that plays out

          7        in terms of time-to-market decisions,

          8        cost-to-market decisions, and cost to the

          9        jurisdiction.

         10             And the one thing that we know is that

         11        they're always unintended consequences, so

         12        trying to think through concurrently, not only

         13        the rigger of the proposed VVSG, but also the

         14        implications of the innovation class model, I

         15        think, has added a level of complexity, at

         16        least in my own analysis.

         17             John first, and then Costis.

         18             >> WACK:  I thought I would just talk a

         19        little bit about what is in the VVSG, and as

         20        best I understand it, why it turned out the way

         21        it was.

         22             At a certain point, in writing

         23        requirements for new systems, there was a

         24        decision made by members of the TGDC that we

         25        couldn't write requirements that were specific



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                156





          1        enough, good enough, for some emerging

          2        technologies, such as cryptographic systems, so

          3        there was a desire since these systems, in

          4        essence, looked good, to somehow or the other

          5        create a pathway to getting them certified,

          6        even though there aren't currently requirements

          7        in the VVSG.

          8             So that's how the innovation class came

          9        about.  And then people struggled for a long

         10        time about, okay, how do we still write

         11        specific requirements for taking systems that

         12        we don't have specific requirements for and get

         13        them through the certification process?

         14             And eventually people started to look at

         15        it as a standards maintenance issue, which

         16        means we've got a problem here, and that is,

         17        how do we update the VVSG in a sense so that

         18        it -- new requirements for new types of systems

         19        can be added to it?  And what can we write

         20        currently that is specific, that labs, again,

         21        can use?  Because labs need very specific

         22        requirements that they can test to.  So the

         23        innovation class was born in a sense.

         24             And to a large degree, a lot of those

         25        questions up there are really directed toward



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                157





          1        how can the E.A.C. do this?  Because the VVSG

          2        doesn't report to do this.  The VVSG has some

          3        requirements in there that essentially say

          4        innovative systems should meet all of the other

          5        requirements in the VVSG.

          6             But the E.A.C., in a sense, was tasked by

          7        the TGDC -- or was recommended by the TGDC to

          8        come up with a system for actually looking at

          9        these new technologies and figuring out how to

         10        get them certified and tested.

         11             So, to a certain extent, I think it is --

         12        I hate to put it this way, but it is the

         13        E.A.C.'s problem, at least that is the way the

         14        TGDC kind of drafted the VVSG.  And they do

         15        need help.  It is a very difficult problem.

         16             I think we all felt that how could we

         17        actually do this within the constraints of a

         18        standard and felt that we could not, that it

         19        had to be a separate process from the standard

         20        that might feedback into it at some point.

         21             >> KING:  Thank you.

         22             Costis.

         23             >> TOREGUS:  Two or three thoughts here.

         24        First of all, I have to agree with my colleague

         25        to the left, that if the VVSG was, indeed,



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                158





          1        performance based, we wouldn't need a

          2        subordinate limited standard group.

          3             And, in fact, from the discussion that we

          4        had last week at the IPI roundtable on this

          5        very topic, the question was raised about a

          6        600-page document.  And in my mind, performance

          7        standards, you don't need 600 pages to describe

          8        performance standards.  You can describe them

          9        fairly adequately in a small number pages and

         10        then challenge innovation to occur.

         11             Some people would almost have you think

         12        that innovation and certification are an

         13        oxymoron put together, that innovation somehow

         14        goes beyond traditional certification, and then

         15        certification catches up to it, and you

         16        certainly don't want to slow down that process

         17        of innovation.

         18             But given that you've got an innovation

         19        class out there, I just wanted to put the other

         20        side of the coin.

         21             In my own state of Maryland, we have a

         22        $1.6 billion budget deficit coming up.  In my

         23        own County of Montgomery, we have $400 million

         24        of a budget deficit.

         25             If somebody even understood there is talk



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                159





          1        about changing the machines we use for voting

          2        to get it one more time, I think people would

          3        just freeze.  They just wouldn't be able to

          4        rationalize it.

          5             So instead of an innovation class, maybe

          6        we also ought to think about some kind of a,

          7        let's say, transformation class.

          8             Given the machines that we have today,

          9        what is it that we can do to the systems or on

         10        the machines, the software inside the machines,

         11        so in fact, when they become more parallel to

         12        what the VVSG would have us think.

         13             I think the notion that we're designing

         14        from scratch systems has to be balanced with

         15        the notion of where are we budgetarily in the

         16        states and in the counties.  We can't just say

         17        we imagine a new system to come into play,

         18        especially since we just had a new system come

         19        into play.

         20             So in addition to the innovative systems,

         21        I would like to put in a good word for how do

         22        we transfer the existing systems we have to

         23        become VVSG compliant so in some kind of

         24        transformative systems, rather than only

         25        innovation systems.  It might be an idea that



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                160





          1        we could have some fun talking about.

          2             >> KING:  Okay.  Juan?

          3             >> GILBERT:  In the innovation class, it

          4        talks about -- well, one of the things that I

          5        think is important is when you submit for the

          6        innovation class -- I guess no one has done

          7        this, so I am assuming you have to submit

          8        material stating why you're innovative.

          9             What is important to me is who is going to

         10        review that.  It goes back to what I said

         11        earlier, about this balance of usability and

         12        security.

         13             I think it is critical that whoever is

         14        reviewing innovation, it has to be a diverse

         15        board or a review committee.  You can't have

         16        all security people on that committee making

         17        decisions because they like a certain

         18        technology that is secure to them, and then no

         19        one in the world can use it without a college

         20        degree in mathematics.

         21             So I think it is important that -- I like

         22        the idea of this innovation class, and I think

         23        certain things, if we're going to have an

         24        innovation class and certify a system, we may,

         25        going back to what he was talking about, is



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                161





          1        maybe certify aspects or methods or components

          2        of systems that could be integrated.

          3             So if you're going to -- for example, a

          4        multi-modal interface, there is clear

          5        usability, performance benchmarks in the VVSG.

          6             If you can meet those, then could you

          7        potentially certify that as an innovation,

          8        because it is clearly something that is

          9        innovative that could be attached to existing

         10        systems, or do you have to certify the whole

         11        system?

         12             And that -- I don't know the answer to

         13        that, but I am throwing it out there, is that

         14        do you provide component certification, or is

         15        there a whole system?

         16             In either case, I think it has to be

         17        carefully decided on who that review committee

         18        is composed of and their backgrounds such that

         19        we get an optimal decision.

         20             >> KING:  Okay.  Thank you.

         21             Brian?

         22             >> HANCOCK:  Thanks, Merle.  I want to

         23        piggyback what Costis said and what Juan was

         24        talking about.

         25             You know, there are a lots of things we



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                162





          1        don't know about what is going to happen with

          2        the next iteration of the VVSG, but I think

          3        there is a couple of things that we certainly

          4        do know and I think probably all of us here can

          5        agree on.  Looking at the document as it is, I

          6        think there are things that everyone needs to

          7        be aware of, especially our policy makers,

          8        legislators and people that are going to be

          9        making decisions at that level.

         10             First of all, whatever this comes out

         11        looking like, the testing is going to be

         12        exponentially more expensive than it is

         13        currently.  I don't think there is any way

         14        around that, given the way it is right now.

         15        Second of all, pretty obvious that no system

         16        out there, and I would include Op Scan systems

         17        could meet this document as it is currently

         18        written.  Those are two things that we know and

         19        I think we need to throw out there for

         20        everybody to think about.  Okay.  Daniel.

         21             >> CASTRO:  I just wanted to expound a

         22        little bit on what Juan said.  I think that is

         23        a very important point because innovations have

         24        historically been good for voting systems and

         25        voting technology, but the idea is that



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                163





          1        innovations can be small and large and can be

          2        very incremental.  But the cumulative effect

          3        often that is what we are really after

          4        sometimes.

          5             It is not clear from the current

          6        innovation class standard which I think the

          7        idea of an innovation class is good in the

          8        sense that it recognizing, that the VVSG is

          9        recognizing the importance of innovation but

         10        bad in the execution of it.

         11             But I don't think it gives a clear path to

         12        certification, especially for small

         13        innovations.  You may have a very small

         14        innovation and a lot of people question, is it

         15        worth going through a very complicated

         16        innovation class certification process?  But

         17        that small innovation may still be very useful.

         18        It may be a minor cost savings, but these many

         19        minor cost savings would add up.

         20             The second point I want to make is that --

         21        I think we talked about this a little bit -- is

         22        that the innovation class is currently defined

         23        as really a subcomponent of software

         24        independence.  That is the other method or

         25        means by which a vendor can get software



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                164





          1        independence.  But again, when we talk about

          2        functional standards versus design standards,

          3        like I have said before, I think software

          4        independence is really design standard.  If you

          5        want to have true innovation, you have to make

          6        innovation class purely functional.  It should

          7        be outside of any kind of design standard.  In

          8        this case, it should be outside of the software

          9        independence class.

         10             >> KING:  I have a comment that really

         11        tries to tie together three different groups of

         12        people who are indirectly mentioned in both the

         13        conversation today but also in the VVSG.

         14             I kind of put this under the heading of

         15        the unintended consequence.  And it is the

         16        individuals involved in open-ended

         17        vulnerability test, the individuals involved in

         18        reviewing innovation class proposals, and the

         19        individuals involved in risk assessment of

         20        systems.  And what is implied is that there

         21        will be individuals who possess some skill and

         22        some knowledge and perhaps some experience for

         23        that.  But I think what we have heard from Juan

         24        and heard whoever made the comment on the OEVT

         25        that the success of those components of the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                165





          1        VVSG will be in large part dependent upon the

          2        abilities and skill sets of those people.  And

          3        I am not proposing that that be something that

          4        we resolve here today.

          5             But I pointed out as for most of us who

          6        are involved in managing organizations,

          7        staffing is the absolute hardest thing that we

          8        do, identifying people who have the skill sets,

          9        the time, and the availability.  And so when I

         10        read through the VVSG that is one of things

         11        that struck me is who are these people?  Where

         12        do we find them?  How do we train them?  How do

         13        we certify them?  How do we supervise them?

         14        How do we evaluate them?  And again under that

         15        heading of unintended consequence, that would

         16        be instructive to better understand in terms of

         17        assessing the doability of the innovation class

         18        and also the open-ended vulnerability testing.

         19             >> GILBERT:  Yeah, we need a functional

         20        spec for the selection of the individuals.

         21             >> KING:  Well, I bring it up because the

         22        intersection of people who understand voting

         23        systems and security, etcetera, it is still a

         24        relatively small number of people.  And I think

         25        we need to pay attention to not only the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                166





          1        unfunded mandates but in this case almost the

          2        human resource mandate that is imbedded into

          3        the VVSG.  John.

          4             >> WACK:  Well, I will not be working in

          5        this forever and I do need a job after this

          6        one.  Probably the rest of us we could form a

          7        corporation here.  But I agree completely with

          8        you.  And at the same time though I do have to

          9        mention that right now NIST has this national

         10        voluntary laboratory system program and part of

         11        that program really is about making sure that

         12        labs, whatever they test, have individuals with

         13        sufficient knowledge and experience and things

         14        of that sort.

         15             I could see that the criteria for voting

         16        system labs will need to be updated somewhat.

         17        There will probably have to be more work done

         18        with that particular program to make sure that

         19        the right level of experience does get in

         20        there.

         21             >> KING:  Sorry.  Ron.

         22             >> RIVEST:  A couple of things with regard

         23        to innovation.  I think one thing that hasn't

         24        been said and maybe is just obvious is the

         25        importance of supporting innovation in this



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                167





          1        area.  It is an area where we all realize that

          2        we are still not where we want to be with

          3        voting systems.  I think academics are

          4        realizing that the voting systems are an

          5        interesting problem area to work in.

          6             We are seeing more and more academics

          7        trying to put their students to working, trying

          8        to come up with better voting systems and

          9        improve it.  We are seeing innovation in part

         10        because of the attention getting and also

         11        because it is a hard problem.  Voting is really

         12        one of very toughest problems I have ever seen

         13        in the security area.  The requirements are

         14        contradictory.  People get challenged by that.

         15             So we are seeing people come out of

         16        academia or out of the wood work, whatever, and

         17        working on this as you might not otherwise

         18        expect.  So innovation is happening in this

         19        area.  The question is how to integrate that

         20        into the certification process.

         21             Juan said something I would like to

         22        support as well.  Is it something on the TGDC

         23        we were not able to do is which was to support

         24        certification components of voting systems.

         25        And I think that has the potential, it is a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                168





          1        hard issue how do that well, but trying to

          2        support certification of the components could

          3        open the door to a lot of interesting

          4        innovation.  If you have an Op Scan system

          5        which was just the scanner part and couldn't

          6        talk through a standardized protocol to a

          7        tabulation system in itself was a separately

          8        certified piece and interfaced standards that

          9        were supported, I think it would open a door to

         10        a lot more innovation.  But I think we are a

         11        ways from that kind of vision yet and making

         12        component certification work.

         13             I think when we are doing new things, and

         14        I think some of the cryptographic systems are

         15        very interesting new proposals, and how to

         16        think about certification.  I think what John

         17        is saying is right, too.

         18             We can have a more open process, a

         19        somewhat higher standard kind of thing.  You

         20        can make it more public and might even require

         21        source code to be public or do other things,

         22        have a lot of public hearings on innovative

         23        ideas, so that it is a different kind of

         24        process in a more transparent way.  And this

         25        may help increase confidence.  You're not



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                169





          1        getting unintended consequences, as Merle is

          2        concerned about.  That's it.

          3             >> KING:  Juan?

          4             >> GILBERT:  I would just like to make a

          5        comment about this, going back to something

          6        Merle said a while ago in some meeting we had.

          7             When you think about what we're doing,

          8        we're talking about election science, and there

          9        is really no discipline -- can anybody go and

         10        major in election science?

         11             I mean, how do you learn how do this kind

         12        of stuff?  So from my perspective, and Merle

         13        knows I always go on the bandwagon about this,

         14        we're talking about election science, and it is

         15        not rocket science.  It is harder.

         16             At least rocket science is a defined

         17        field.  At least you can get training in it.

         18        At least there is something there.

         19             Here, what do we have?  We have

         20        disciplines that have traditionally been in

         21        silos, you know, separate disciplines that are

         22        coming together trying to do something that has

         23        never been done before, and we really don't

         24        have true training on how to do this.  So we're

         25        kind of making it up as we go along in one



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                170





          1        sense, but I think we are kind of hard on

          2        ourselves, too.

          3             So I think the VVSG -- and maybe this

          4        could have been a summary -- but I think the

          5        VVSG is a step in the right direction, and

          6        we're definitely critical of it.

          7             But, again, keep in mind that there really

          8        isn't a discipline or a book for dummies on how

          9        to do this stuff, so...

         10             >> KING:  Juan makes a good point.  I just

         11        saw where NASA has postponed the shuttle launch

         12        for a month because of a failed fuel cell

         13        sensor.

         14             And if we have bad fuel cell sensors on

         15        our election equipment, we are still going to

         16        have an election on election day, so maybe it

         17        is a lot more difficult than rocket science.

         18             Peter?

         19             >> RYAN:  Yes, a comment that Ron just

         20        made that prompted in my mind, that moving to,

         21        say, cryptographic systems, that potentially

         22        changes the certification game, because it

         23        almost then becomes a much more public process.

         24             It's much more transparent.  What you are

         25        trying to evaluate there are the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                171





          1        crypto-algorithms, the protocols, and I guess,

          2        to some extent, the procedures around them and

          3        so forth, rather than trying to certify a lump

          4        of code.

          5             And that is, in a sense, precisely the

          6        point of making the thing software independent,

          7        is that it doesn't depend on certifying a lump

          8        of code.  So in some sense, you're changing the

          9        certification game there, and potentially

         10        making it perhaps easier and cheaper, perhaps.

         11             Just to come back, you made some comment,

         12        I think, that the innovation class is likely to

         13        make things exponentially more expensive to

         14        certify, or something to that effect.

         15             >> Just the VVSG [indiscernible]

         16             >> KING:  Okay.  Alec?

         17             >> YASINSAC:  I think one of the things we

         18        haven't really addressed.  Peter, you just

         19        raised a perfect issue.  And while it may be

         20        more transparent in terms of everything is out

         21        there in the open, I think what it would mean

         22        is very, very far fewer people would be able to

         23        have any real idea of what was going on.

         24             I mean, if you look at the number of

         25        programmers there are in this country that



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                172





          1        really understand how to write C plus plus,

          2        that is a small percentage of the population,

          3        but if you look at the percentage of people who

          4        could understand most of these algorithms, it

          5        is extremely narrow and extremely small, which

          6        is one of the large challenges of that

          7        solution -- of those set of solutions, I should

          8        say.

          9             >> KING:  Ron?

         10             >> RIVEST:  I would like to respond to

         11        that issue.  It is one that is raised often

         12        with cryptographic systems.  And cryptography

         13        is the use of mathematics for integrity and

         14        secrecy.  It is a technology that has been

         15        around for quite a while.  It's taught at lots

         16        of universities.

         17             And in some ways, it's much simpler than

         18        software.  So if your choice is understanding a

         19        few equations and understanding 10,000 pages of

         20        code, you know, I think the choice is clear as

         21        to which one is more digestible.  In fact, the

         22        code isn't even available to look at here.

         23             So I think it is a bit of a false issue

         24        here.  When you think about how people develop

         25        confidence in systems.  Some understand it



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                173





          1        themselves, and I think ones that are publicly

          2        documented with technical articles that you can

          3        read and digest that will be popularized for

          4        this stuff, too, will help with that.

          5             But not everybody understands all of the

          6        technical details of any system.  You certainly

          7        don't understand how an airplane works except

          8        in the grossest terms.  Certainly people use

          9        D.R.E.'s and D.R.E.'s with VVPATs don't

         10        understand how the software works on that,

         11        either.

         12             So I think you have to be fair when you

         13        are talking about what it takes to

         14        understand -- there has to be some social

         15        process, whereby we as a society get

         16        comfortable with the systems we're using.

         17             And that is going to involve experts at

         18        all levels of applying their expertise,

         19        communicating with people who are good at

         20        explaining things, having everybody -- making

         21        the system as public, open, and transparent as

         22        possible and getting everybody a level of

         23        comfort with it.

         24             And I don't think that cryptographic

         25        systems are any different in meeting that kind



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                174





          1        of a process than our D.R.E.'s or something

          2        like that.  We have been through that process

          3        and we have seen that they have some problems,

          4        so getting digested in that way.

          5             So I think we have a process in place for

          6        taking complex technology and making it

          7        something that people are comfortable with.

          8             All of the people that have the expertise

          9        have looked at this and they feel okay with it.

         10        Many people say, okay, I'll go along with that.

         11             It doesn't need to be the case that

         12        everybody needs to understand every aspect.

         13        Even a paper ballot system, not everybody

         14        understands how a paper ballot system works.

         15        They may not understand what a provisional

         16        ballot is.  They may not understand what the

         17        counting procedure is like or how they're

         18        transported to the county office.

         19             There is lots of details that people just

         20        assume get taken care of.  You have to think

         21        about a broader scale what it takes for

         22        technology to be accepted in a social manner.

         23             It is not easy.  I am not saying that

         24        anybody should take a cryptographic proposal

         25        and accept it blindly, because there may be



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                175





          1        mistakes there, too, or other problems to be

          2        dealt with, but just to have a larger

          3        perspective on what the process is, take

          4        technology, and integrate it into our lives.

          5             >> YASINSAC:  And if I may.

          6             >> KING:  Go ahead.

          7             >> YASINSAC:  I agree that the general

          8        population is probably not a ton of difference,

          9        but the question -- the point was, the ability

         10        to have the transparency and the openness.  It

         11        is like, you know, if the code were all written

         12        in Russian, I couldn't tell anything about it.

         13             If it is written in cryptography, that

         14        just makes it a much smaller set of people that

         15        are able to understand that technology and

         16        analyze it in detail, in my view.

         17             Of course, which brings up a point, that,

         18        you know, when I was in the marines, it was

         19        every marine was a rifleman.  In school, every

         20        student, I believe, should be a cryptographer,

         21        so that's what I am pushing for.

         22             >> KING:  I am going to think about that.

         23             I have a question for my colleagues in the

         24        elections community.

         25             The term "innovation" is pregnant with



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                176





          1        meaning, and I think most election people have

          2        had about all of the innovation they can

          3        stomach.

          4             In fact, the word "stability",

          5        "predictability," "stasis," all have a more

          6        positive connotation in the election community.

          7             So from your perspective, could you

          8        comment on the need for the VVSG to accommodate

          9        innovation and how that factors into your

         10        vision for your unit's use of technology in the

         11        foreseeable future?

         12             >> THOMAS:  I think we would all like to

         13        see a better mousetrap, okay.  I don't think

         14        there is any election official out there who

         15        would say, what we've got right now is perfect.

         16        But they will say, in many instances, what we

         17        have right now, we know how to operate and make

         18        it work.

         19             And innovation recently hasn't worked so

         20        well.  And the -- the cases are obvious out

         21        there in terms of problems that people have

         22        had.  In a broader political realm, not just

         23        partisan politics, but election technology is

         24        being rolled out in a very different

         25        environment than it ever has in the past, and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                177





          1        much of which doesn't have a lot to do with the

          2        Florida situation in the 2000 election.

          3             So given all of that, I guess we would

          4        prefer that you not call it innovation, that

          5        you find some other phraseology, and I also go

          6        back to the issue about components.

          7             And I find this as an irritant to me,

          8        because quite frankly, we're developing things

          9        with our voter registration system that are

         10        going to intersect with the voting system, and

         11        it is going to gather candidates from across

         12        the state and give local officials the ability

         13        to roll that into the existing software.

         14             Now, I cannot be dependent on some vendor,

         15        perhaps against their financial interest, to

         16        take that forward and infinitesimal, and this

         17        system doesn't really give me an option.

         18             It gives me an option to either not to

         19        advance or to take what is put on the chopping

         20        block and be happy with it, and we are going to

         21        move ahead regardless.

         22             And so I find that to be a shortcoming,

         23        that there is not some type of a component

         24        certification.  And quite frankly, I view the

         25        auto market as a component.  It is an electric



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                178





          1        pencil.

          2             So, yes, we would like to see it move

          3        forward.  We would like to see innovations.  I

          4        don't know who the first one is that is going

          5        to step up to buy this next system that is

          6        cryptography; somebody will.

          7             But I think what is happening now -- and

          8        you're seeing it as jurisdictions move to

          9        optical scan -- is that there is -- it is a

         10        fallback.  You're going back to something you

         11        know how to operate.

         12             The voter gets it.  I mean they do

         13        understand much like the lever machines, I pull

         14        a lever down, a little wheel turns.  I did have

         15        one voter once say they know how I voted just

         16        by turning those wheels backwards.  But that

         17        was only one out of several million.  So I do

         18        think there is a point at which this does get

         19        beyond the comprehension of the voter in terms

         20        of what is this thing that I am now being asked

         21        to trust to take that leap of faith and there

         22        is a lot of competing interest and competing

         23        messages that talk about whether it is good or

         24        bad or you should trust it or they can figure

         25        out how you wrote it.  And that wasn't present



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                179





          1        many years ago.

          2             That is new.  It is there.  That is

          3        community that obviously is at the table that

          4        everyone has to deal with.  But I don't

          5        think -- I am not saying people are satisfied

          6        with the status quo.  I don't think election

          7        officials are.  Clearly they are not.

          8             But I will just leave this as my summary.

          9        Where these new guidelines go is what is the

         10        unintended consequence is that there will not

         11        be a system that anyone can afford to buy, that

         12        is going to be produced.  You know, we bought

         13        systems recently with federal money.  We spent

         14        $65 million for both of our components, and we

         15        are told that these things are going to last

         16        about seven years.  And there is no $65 million

         17        unless Congress is coming back to the trough

         18        for us that is going to available in seven

         19        years to go out and buy another one of these

         20        systems.

         21             So I do see a lot of things coming

         22        together.  So if the innovation is cheap, it

         23        will have great merit as part of it.  But if it

         24        is going to be another one of these huge cost

         25        increases to state and local government, you



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                180





          1        are going to see a lot more Optical Scan out

          2        there.  With the, you know, the issues at hand,

          3        we are Op Scan state wide.  We can make it

          4        work.  I would never tell you it is perfect by

          5        any means, but we can make it work.

          6             >> KING:  Sarah.

          7             >> JOHNSON:  Just to add a little humor to

          8        the meeting.  We are all so serious here.  This

          9        is my own personal opinion here.  I tend to

         10        feel like election officials are the tennis

         11        ball in a tennis match.  We are the ones being

         12        batted back and forth across the net, back and

         13        forth activist groups, you know, Congress, our

         14        own state legislators and where we are the ball

         15        being batted back and forth.  And we just want

         16        people to leave us alone.  For a little while.

         17        Five minutes.  So that we can shake our head

         18        and go, woo, okay.  Or have some of Kentucky's

         19        finest beverage.  How about that?

         20             Actually, we two agree and I agree, and I

         21        think the Standards Board agree that we don't

         22        like the term innovations especially as wide

         23        open as it is.  And we will be commenting on

         24        that in a couple of days.

         25             I agree with Chris, we don't specifically



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                181





          1        like the voting machines that we have now.

          2        There are things we would like to fix from a

          3        procedural standpoint, the usability

          4        standpoint, etcetera.  But as Chris said, we

          5        just spent $21 million buying new systems to

          6        meet the federal requirements, and we don't

          7        have that kind of money sitting around to

          8        instantly replace these.  So it is a concern

          9        for all of us.  And any time you are talking

         10        about voting equipment, as some of the other

         11        panelists have mentioned, it might be from a

         12        technology side the best thing in the world.

         13        It might be the best thing that can you come

         14        up.

         15             It might be brilliant and we might even

         16        agree with you if we understood everything

         17        about it, that it is brilliant.  But we have to

         18        then know how to use it ourselves, bring it

         19        back home, teach our county clerks as I said

         20        earlier, teach the precinct officers, mind you

         21        who are still reeling from the fact of going to

         22        these newer audio ballots and kind of more

         23        computerized, the average election official is

         24        in their 70s and they are still trying to

         25        understand that knob thing that you have to



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                182





          1        turn on those voting machines, as they call it.

          2             Let alone the voter.  So we have a lot of

          3        things that factor into that.  All of that

          4        dependent upon Congress or the state

          5        legislature for the funding for that issue.  So

          6        while we appreciate innovation and never want

          7        to stifle it, if whatever it ends up being, it

          8        has got to be a piece of a much, much larger

          9        puzzle.

         10             >> KING:  Thank you.  Peter.

         11             >> RYAN:  Yes, I just had a couple of

         12        quick comments.  One -- first one is I don't

         13        think innovation technologies in this field

         14        don't have to be expensive.  I think I would

         15        suggest the one that I have been working on can

         16        run on quite conventional off-the-shelf kit so

         17        in that sense it could be quite inexpensive.

         18             I want to switch to another thing and

         19        question really to you guys.  The certification

         20        process in the U.S., who pays for it at the

         21        moment?  Is it paid for by the vendors or?

         22        Vendors.  And that is likely to stay like that?

         23        Because I saw -- I wonder where there might be

         24        a merit in considering some sort of hybrid

         25        thing where may be -- I don't know the vendors



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                183





          1        perhaps pay for initial evaluation and

          2        feasibility and to help the innovation may be

          3        then, it could be subsidized a bit by federal

          4        funds or something.  That might be a thought.

          5             Just one other thing I wanted to say,

          6        certainly in the UK for evaluating secure

          7        systems in general, not voting systems

          8        specifically, there are methods for doing that,

          9        so there are so called clefts, CSB licensed

         10        evaluation facilities, you have probably heard

         11        of them.  So CSG is sort of an analog of NCIS I

         12        guess it used to be called.  Center for

         13        computer security, or something.  Well, they

         14        proclaim themselves as the national authority

         15        on information assurance systems.  They license

         16        companies like logic to provide these

         17        evaluation facilities.  So companies come to

         18        them, and the person who certifies the

         19        certifiers is CSG in the context.

         20             I don't know if you have any analogous

         21        model in the U.S. or if it is worth checking

         22        into.  And of course behind all this is the

         23        framework for evaluation and I know some people

         24        wondered to what extent the common criteria

         25        will be applicable to voting technology.  I am



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                184





          1        not sure that question has ever really been

          2        answered.  My own suspicion is not entirely but

          3        maybe there is some useful ideas that can be

          4        carried across.  So just a few comments.

          5             >> KING:  Thank you.  Daniel.

          6             >> CASTRO:  Thanks.  To respond to your

          7        question about who pays for the certification.

          8        I mean the vendors pay.  I argue it is really

          9        the tax payers who are ultimately paying.  That

         10        is the matter in the federal government can pay

         11        for it, the vendor can pay for it and the

         12        federal government can pay for it when they

         13        give the money to the states who will buy the

         14        voting systems.  The other point I just wanted

         15        to respond just because I work for the

         16        Information Technology and Innovation

         17        Foundation so I hate that innovation is going

         18        to get batted around.

         19             So the definitions of innovation that most

         20        people use is new and better.  So I think there

         21        is a lot of time where we talk about something

         22        that is new but not necessarily better.  So I

         23        argue that is not innovation.  And in that

         24        sense, let's not call that innovation.  Let's

         25        call it new technology which is more costly.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                185





          1             >> RYAN:  Cheaper too.

          2             >> CASTRO:  And more cheaper.

          3             >> KING:  I would like to make a comment

          4        about Daniel's observation about who pays for

          5        testing.  From the vendors' point of view that

          6        is all cost recovering and then the question

          7        becomes from whom do they recover the cost?

          8        And it can be from the innovators and so if

          9        there is a small cadre of jurisdictions who

         10        rally behind a modification, then they will

         11        bear the cost of that testing over whatever

         12        time arise in the vendor chooses to spread it.

         13             So it is a very difficult decision I am

         14        sure for the vendors to make about how far they

         15        spread the cost and to whom.  But I think you

         16        are correct that ultimately it is the

         17        jurisdictions and the taxpayers in those

         18        jurisdictions who pay for the cost of the

         19        testing.

         20             >> KING:  Costis.

         21             >> TOREGAS:  Again, drawing from other

         22        sectors, the whole notion of the new technology

         23        coming to government is very well-known and has

         24        been studied extensively.  And the key question

         25        to ask is who drives it into the government?



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                186





          1        In the election technology field, if I am not

          2        mistaken, it is driven by legislative might,

          3        there is a law that is passed that says this is

          4        the way we will do it.  And that has unintended

          5        consequences for sure.

          6             In order to kind of bring back some kind

          7        of a more rational approach to it, I continue

          8        to think that some kind of a cost benefit

          9        analysis of either the standard or irrespective

         10        of the standard, the whole process of

         11        innovation in election technology would be

         12        extremely helpful.  I think the vast majority

         13        of elected officials who end up having to vote

         14        for the budgets, whether state legislators or

         15        county commissioners, they have to vote to

         16        appropriate money or use money that actually

         17        goes back to the taxpayer argument.  Have to

         18        begin 106 the principles behind the innovation,

         19        and you are absolutely right, innovation

         20        shouldn't drive you towards better

         21        circumstances.

         22             But the question is, according to whose

         23        standard?  And most of the times, the

         24        technology is driven by the user component.

         25        This is an unusual one where the user is not



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                187





          1        necessarily in the driver's seat.  If you stop

          2        and think about enterprise research planning

          3        systems, you stop to think about CRM systems,

          4        you stop to think about inventory systems,

          5        those are traditionally driven by the user

          6        community because they are the ones that reach

          7        into their wallet, pay, and they know what they

          8        want.  In this particular circumstance, we have

          9        a variance from that innovation transfer

         10        mechanism so it might be useful to think about

         11        how to handle innovation when it is being

         12        driven not by the user, the end user.

         13             >> KING:  Peter, is your sign up?

         14             >> RYAN:  Sorry.

         15             >> KING:  Well, now my mic is back on.

         16             Are there any more comments on this

         17        question number five?

         18             Okay.  What I would like to do next is

         19        kind of begin the wrap up of this -- of this

         20        panel discussion.

         21             And what we have talked about is that each

         22        panelist would be given an opportunity to kind

         23        of make a summarizing statement, if there is

         24        things that you feel are important for your

         25        colleagues on the panel or members of the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                188





          1        audience or the Standards Board folks to take

          2        away from this.  This is the opportunity to do

          3        that.

          4             As I am looking through my notes today, we

          5        have covered a lot of ground.  We have talked

          6        about a lot of different things from the

          7        perspective of the testing labs, from the

          8        perspective of NIST, from the academic point of

          9        view, from the election official point of view,

         10        so I think we have done a good job in terms of

         11        the scope of what we have covered.

         12             What I would like to do -- and, Ron, since

         13        we started with you today at the opening, I

         14        would like to give you the last word in the

         15        closing.  I think that is fair.

         16             So I would like to start in this direction

         17        and kind of work around the table and, again,

         18        think about what you heard today.  Think about

         19        the message that you brought about the things

         20        that are important for continued discussion and

         21        continued reflection.

         22             And then I would like to make a few

         23        closing comments, and then we're going to let

         24        the E.A.C. close up the session today.

         25             It has been a long day and I am -- I don't



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                189





          1        think I can go much beyond 6 o'clock here.  So,

          2        Juan, if we can start with you, and then we'll

          3        move around the table.

          4             >> GILBERT:  I am very brief.  I think the

          5        VVSG is good.  I like the innovation class.  I

          6        think we need to test it.  I think we need to

          7        put some systems through the innovation class,

          8        and let's see what happens.  The innovation

          9        class, in itself, is innovative.  Let's

         10        evaluate it.

         11             I would like to reiterate my comment, that

         12        election science is not rocket science.  It is

         13        harder.  I also would like to say that you have

         14        to have multiple disciplines at the table for

         15        this because it is an interdisciplinary

         16        approach to a very hard problem.

         17             And we have got to be careful about who is

         18        doing the evaluation.  We need a diverse team

         19        that brings different things to the table to

         20        ensure that we cover all aspects necessary to

         21        get the best system or process in place for our

         22        citizens.  And I end there.

         23             >> KING:  Thank you, Juan.

         24             Mike?

         25             >> SHAMOS:  Yes.  There is an elephant in



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                190





          1        the room, he was introduced by Brian, but I am

          2        going to flesh him out, and that has to do with

          3        the cost of building a system that will comply

          4        with the 2007 VVSG and the cost of testing a

          5        system to determine whether it satisfies the

          6        2007 VVSG.

          7             In 1980 Pennsylvania passed its electronic

          8        voting statute that laid out 17 mandatory

          9        requirements that any voting system had to

         10        satisfy.  That list of 17 has not been amended

         11        since then, and we're now 27 years later.

         12             That set of 17 requirements was sufficient

         13        for punch-card systems, Optical Scan systems,

         14        D.R.E.'s without paper trails, D.R.E.'s with

         15        paper trails, and some of the systems that,

         16        today, we're classifying as innovative.

         17             Those 17 requirements can be written in

         18        two pages.  I have them in front of me right

         19        here.  I am not even going to pretend that

         20        they're adequate.

         21             However, we now look at the 2007 VVSG at

         22        600 pages.  That's 300 times more requirements

         23        than have been satisfactory for the state of

         24        Pennsylvania, the sixth largest state in the

         25        country, for 27 years.  We haven't had a



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                191





          1        serious challenge to the security of an

          2        election in Pennsylvania during that time.

          3             In those days, the vendors were able to

          4        get a certification exam by paying $450.  That

          5        was too cheap.  The price was set in 1929, and

          6        wasn't modified since then.

          7             I have no idea how much it is going to

          8        cost a vendor to go through a 2007 VVSG

          9        certification, and I think, at some point,

         10        we're going to have to face that question

         11        directly.

         12             There is going to have to be a

         13        subcommittee of somebody who figures out what

         14        it is going to cost and who is going to pay for

         15        it.  That is, I think, the major unspoken thing

         16        from today.

         17             >> KING:  Thank you, Mike.

         18             Costis.

         19             >> TOREGUS:  I'll try to be equally brief.

         20             I, too, was very nervous, when I heard

         21        Bryan's two comments.  They're shattering to

         22        me.  That means we need to take stock in the

         23        VVSG and make some changes.  We cannot continue

         24        on the path given the two statements that were

         25        made.  Something has to happen differently.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                192





          1             My contribution to the discussion has to

          2        do with two different things.  One is, I

          3        believe that something again -- perhaps another

          4        component of the E.A.C.'s handling is a strong

          5        communication strategy, especially towards

          6        elected officials, the both fronts of election

          7        technologies, for simple things like risk

          8        assessment and simple things you can't have 100

          9        percent guarantee.

         10             There are ways to begin to change the

         11        mindset of people.  You cannot change the

         12        mindset of people without trying, and I think

         13        all too often, we accept the fact that someone

         14        in the state legislature or someone in the

         15        county council will not understand that you

         16        can't have 100 percent of something, and I

         17        think a strong communication strategy can do

         18        that.

         19             The second point is going back to your

         20        comment about vocabulary -- building vocabulary

         21        that we can begin to communicate with.  Beyond

         22        the vocabulary, we need to have simultaneous

         23        dialogues of the stakeholders.

         24             It is not healthy to have stakeholders in

         25        this room and stakeholders in the next room and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                193





          1        the next room.  The mathematicians will tell us

          2        if we optimize subparts and put them together,

          3        that is a suboptimal solution.

          4             In order to find the optimal solution, we

          5        have to do it with a simultaneous discussion,

          6        and perhaps that is beginning to happen, and I

          7        look forward to hearing more and to participate

          8        more in this.

          9             And then the final concept is one of cost.

         10        I think in today's world, cost will drive quite

         11        a lot of things.  I stopped believing in Santa

         12        Claus.  Perhaps Santa Claus comes from time to

         13        time, and we're all too happy to have Santa

         14        Clause come.  But the reality is that we have

         15        what we have.

         16             So, therefore, incremental changes and

         17        making sure that we can cross-connect and

         18        understand from each other's best practices can

         19        be very, very helpful, so I think the VVSG

         20        could be thought about in two parts.

         21             One is, somebody can rip off and apply to

         22        existing systems and another part that will

         23        give us our future vision 10, 20 years out.

         24             Thank you.

         25             >> KING:  Thank you, Costis.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                194





          1             >> John?

          2             >> WACK:  Well, it has been interesting.

          3        I think the VVSG is -- the recommendations has

          4        really touched off a lot of emotions in a lot

          5        of different audiences.  And to a certain

          6        extent, that's good.  To a certain extent, it

          7        works against the fair evaluation of it.

          8             I think that the VVSG needs to be

          9        considered as, first of all, a beta version.

         10        It's going to get two more rounds of review,

         11        and, therefore, I think that it needs to be

         12        read thoroughly and understood and comments

         13        made specifically on that.

         14             The other thing, I think is important, is

         15        that it be considered as a tool.  It's a tool

         16        that was made by the TGDC.  It is a hand plane.

         17        It's a power saw, whatever.  It is a tool

         18        designed to do something.

         19             And there are many aspects to it.  There

         20        is an awful lot of usability on it and stuff, a

         21        lot of reliability stuff.  There's a lot of

         22        security stuff.  It does need to be

         23        considered -- at least I think the authors

         24        intended it to be considered as a whole, if you

         25        were to remove software independence, you might



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                195





          1        need to then all of a sudden take a look at

          2        what is in the reliability area, decide that

          3        needs to be revamped quite a bit.  It needs to

          4        be looked upon as a standard for future voting

          5        systems, not today's voting system.  Today

          6        voting systems work with VVSG 2005.  It is not

          7        intended to, you know, codify probably any

          8        particular type of voting system today,

          9        especially VVPAT.

         10             In no way do I think the authors of VVSG

         11        wanted to promote VVPAT.  It is a big standard.

         12        Don't let the 600 pages scare you.  They -- the

         13        number of pages could have been reduced

         14        significantly but was made to be 600 largely so

         15        it would be more understandable and readable.

         16             Brian, I do have to take a little bit of

         17        issue with your exponential increase in

         18        testing.  Exponentially increasing cost in

         19        testing is a huge increase in testing and

         20        expense, and I don't know that that really will

         21        be true.  Certainly it will be more expensive

         22        but that isn't the VVSG's fault.  I think

         23        that -- I believe the number of states that

         24        have gone towards using all paper is around 38

         25        and there are more to follow.  And people are



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                196





          1        just more concerned about the quality of voting

          2        systems today, and the increase in testing

          3        would occur kind of regardless I think.

          4             With that, I just would like to close and

          5        say I think it is worth examining closely and

          6        making sure that you just don't latch to

          7        specific areas and say, you know, that is not

          8        good enough or that is good.  I think it needs

          9        to be examined in its whole, not just the

         10        security aspects that we mainly discussed

         11        today.  Thank you.

         12             >> KING:  John, thank you.  Brian, I will

         13        skip you and hold you for the last.  Sarah.

         14             >> JOHNSON:  One thing I would like to say

         15        thanks.  I am representing the Standards Board

         16        here and a lot of issues that are our board has

         17        come up with potential draft resolutions have

         18        been discussed today, so it is a little

         19        gratifying from our end as election officials

         20        although we are certainly not your equals in

         21        the technical field.  And likewise in our

         22        election field, it is a two-way street.  It is

         23        really gratifying here that a lot of our issues

         24        were echoed and certainly discussed here.  So

         25        there are a lot of things I will be taking back



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                197





          1        to my group as we meet starting tomorrow to

          2        formulate our formal resolution/comments on the

          3        VVSG.  And I do appreciate that on that behalf.

          4             Just kind of in reference to some other

          5        things, I mean, it is true the VVSG is a tool

          6        and it does have a specific purpose.  But we

          7        all know that through life, tools get used for

          8        thing that they weren't necessarily intended to

          9        be used for.  For example, in college, I found

         10        that a wooden meat tenderizer, my little

         11        utensil thing works great as a hammer.  It

         12        wasn't intended for that, but it works when you

         13        need to hang a picture up.

         14             So I think that that taken in the context

         15        of where we are now, the political realities

         16        that we as election officials deal with, that

         17        the E.A.C. will deal with, does shine somewhat

         18        of a different light on perhaps the VVSG.  It

         19        does say they are voluntary.  Big V, Voluntary

         20        Voting Systems Guidelines.

         21             But in reality, once you take something

         22        that is voluntary and it also does depend a

         23        little on your state law, sometimes they don't

         24        become voluntary.  They may say they are

         25        voluntary, but in reality, they are mandates or



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                198





          1        standards that all state's at some point are

          2        held to that.  And the other issue I would

          3        point out is it is true that this particular

          4        VVSG draft or any iteration, whatever ends up,

          5        is supposed to be for future voting systems.

          6        But again that political reality will make it a

          7        little more difficult to stick with that future

          8        voting systems because once they are in effect,

          9        they very well can be looked at and compared to

         10        current voting systems.

         11             So we do have to deal with elections under

         12        current voting systems even though this

         13        100-pound elephant of called the VVSG for this

         14        new class, whatever you do call them,

         15        technologywise and we all have to live with

         16        that.  That is something we are struggling with

         17        as election officials and federal and state

         18        politicians will often struggle with is the

         19        concept that these are voluntary and that this

         20        is for some future system that doesn't exist

         21        when in reality we are dealing with something

         22        that does exist.

         23             >> KING:  Thank you, Sarah.  Chris.

         24             >> THOMAS:  I would just very briefly

         25        agree with Sarah's comments.  As a community,



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                199





          1        we are very much involved or agree with what

          2        you are bringing to the table.  This is

          3        expanding the horizons of this community pretty

          4        extensively as over the past 30 years.  What

          5        has happened in the last few years has been a

          6        great leap.  And we are still trying to sort

          7        all that out.

          8             So we don't want to leave the impression

          9        that we are trying to keep people out of our

         10        business.  We are not.  We understand that we

         11        have got to operate as an inclusive part of

         12        government and of elections community and we

         13        are willing to do that.  But we are looking to

         14        see where the dust is going to settle.

         15             The fear of, like Brian saying, that even

         16        Optical Scan wouldn't survive under these

         17        standards is a little mind boggling.  So you

         18        know, we will look at this, and I appreciate

         19        all your comments.  I will definitely taking

         20        many of them back to the Standards Board --

         21        Board of Advisors, excuse me, tomorrow.  And

         22        there are a number of folks that are in the

         23        audience here that are also getting the benefit

         24        of it that will be deliberating on these over

         25        the next couple of days.  So thank you.



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                200





          1             >> KING:  Thank you, Chris.  Alec.

          2             >> YASINSAC:  Right.  There are two quick

          3        points that I would like to just reemphasize as

          4        take aways.  And I believe I said the first one

          5        a couple of times.  All software and electronic

          6        voting systems ought to be high assurance and

          7        engineered rigorously.  I believe process

          8        maturity is something that the vendors will

          9        be -- would be willing to take on and could

         10        take on at their own expense and make it part

         11        of their business process.  And then the OEVT

         12        could accomplish the purpose of back door

         13        verification of the process viability through

         14        the assessment of the product.  And

         15        unfortunately, I don't think the VVSG leveraged

         16        and picked up this -- what I think would be a

         17        tremendous improvement.

         18             The second take away I would like to leave

         19        is that one of things I think software

         20        independence pushes us further and further

         21        toward is the auditing on the back end, the

         22        first count not being as important.  And there

         23        is an old adage that says, we got time to do

         24        it, do it over.  But we don't have time to do

         25        it right the first time.  I think we can't do



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                201





          1        that in voting systems.  We have to get the

          2        first count right and that is what it is all

          3        about.  So I hope you will take that away from

          4        today.

          5             Thank you very much.  Thanks for allowing

          6        me to be here.  I am honored and pleased.

          7        Thanks.

          8             >> KING:  Thank you, Alec.  Daniel.

          9             >> CASTRO:  I just wanted to respond to or

         10        highlight some of the things that other people

         11        said.  I would like to extend on the first

         12        thing Ron said earlier, that the goal of the

         13        VVSG is to get demonstratively correct voting

         14        systems.  I think we are talking about the

         15        VVSG, I mean we have said this before, it is

         16        about voting system guidelines but really what

         17        we care about in the end is about the elections

         18        we want demonstratively correct election

         19        systems.  Not just voting systems.

         20             And I think we need to consider that when

         21        we are working with standards, it is not enough

         22        to say it is the voting system right but it is

         23        the entire election right, just remembering

         24        that.  And when we talk about any aspect, not

         25        just security, also cost when you are looking



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                202





          1        at cost.  It is not just about how much does

          2        the voting system cost but how much does it

          3        cost to operate and certify and whether the

          4        costs are involved in the process is there.

          5             And then I also want to mention two things

          6        that Juan said.  First of all, Juan had

          7        mentioned that the innovation class is

          8        innovative.  Maybe so is software independence

          9        but again I want to go back to that definition

         10        of innovation.  It is definitely new, not

         11        necessarily better but that is something that I

         12        think we have been talking about.

         13             Another thing that Juan mentioned is that

         14        usability goes hand in hand with security.  I

         15        think it is very important that we keep in mind

         16        that there is many different values here:

         17        Security, usability, reliable, accessibility,

         18        cost, and the impact on innovation.  And all of

         19        these are values that the VVSG and the E.A.C.

         20        when they are look at the VVSG should be

         21        seeking to improve.  I think we can all agree

         22        that some future voting systems will be

         23        significantly better than what we have today.

         24        And the goal is how -- when we are looking at

         25        VVSG, the VVSG should be asking how do we get



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                203





          1        there and how do we get there sooner?  Rather

          2        than later.

          3             >> KING:  Okay.  Thank you, Daniel.

          4        Peter.

          5             >> RYAN:  Okay.  Well, first of all, I

          6        would like also to say thank you very much for

          7        the invitation to be here.  I am honored and

          8        privileged, and I have really enjoyed it.  I

          9        hope I have contributed something to it.  And I

         10        am very pleased to see the way you are sort of

         11        conducting this in an open transparent way.

         12        There is that transparent word again.  I hope

         13        perhaps something similar will happen in the UK

         14        in Europe.  I am not aware of anything.  I

         15        think you are ahead of game in that sense.  I

         16        guess you have been ahead of the game in voting

         17        technology for over a century now.

         18             The UK generally speaking we are just

         19        using old paper and stubby pencils.  So am

         20        pleased to witness this and to some extent

         21        participate in it.

         22             I would also point out that of course this

         23        is not just a U.S. issue, and we are facing to

         24        some extent, perhaps a lesser extent but to

         25        some extent the same sort of issues over in



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                204





          1        Europe and so on.  So whether there is any

          2        possible synergy there that can be exploited, I

          3        don't know.  But it is worth thinking about

          4        maybe.

          5             I would also like to echo Juan's comments

          6        about and it is highly disciplinary and

          7        sociotechnical and so I think people are now

          8        beginning to appreciate the extent to which

          9        that that is true, and we really do need to

         10        look at it in the large -- and as you said,

         11        look at the interests of the various

         12        stakeholders, and speaking for myself, I would

         13        really welcome the opportunity to be able to

         14        talk to more stakeholders, election officials,

         15        and politicians, as well as voters and so on

         16        and so forth.

         17             And I am also so very pleased to see this

         18        innovation category class that you're

         19        introducing, and whether it's framed quite

         20        right now at the moment is difficult, but I

         21        think it is great you're trying to be open to

         22        innovation.

         23             I certainly think, speaking of some

         24        self-interest here, I guess, but I think there

         25        are innovative technologies out there which



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                205





          1        hold out great promise and that is really being

          2        innovative and potentially better in many

          3        senses for providing higher security, and

          4        usability, reliability, and so forth.  And

          5        potentially, that's even at lower costs.

          6             But of course, all of that has to be

          7        weighed up, and there are still, I think, major

          8        research issues there, which is why I guess I

          9        am sort of slightly hesitant about -- there is

         10        always this problem with new technology:  At

         11        what point do you standardize and what do you

         12        standardize and how do you standardize and so

         13        on?  And you are obviously facing that issue.

         14             I guess I was slightly confused by some of

         15        the distinctions between sort of functional

         16        design and performance standards.  I am not

         17        quite sure I quite understand the distinction

         18        between these.  I suspect actually that the

         19        distinctions get blurred at times.

         20             One of the things I have become

         21        increasingly conscious of over the years, when

         22        I was a young, naive security guy, I thought

         23        that you started off, you put out your security

         24        policies, high-level abstract policies, and

         25        then you sort of trickled down through the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                206





          1        system, and you draw out your threats and so

          2        on.

          3             But when you speak to people in the field,

          4        they say the policies are typically driven by

          5        the threats.  So in fact, it doesn't work in

          6        this sort of top-down way.

          7             And also, there is also a tendency to sort

          8        of confuse properties with mechanisms, which is

          9        another thing which happens a lot in security.

         10        I suspect the same sort of thing may be

         11        happening to some extent here.

         12             So I think a great deal of thought has to

         13        go into how to frame things at this high level

         14        so you get the right degree of constraint but

         15        not overdoing it.  That is going to be very

         16        tricky, and I think it's going to take time and

         17        feedback and so on and so forth.  So I think

         18        that's pretty much what I wanted to say.

         19             I just wanted to throw in, I think you

         20        were saying it's nice have catch phrases you

         21        can latch on to.  One which I heard from Josh

         22        Fenelobe (phonetic) who claims that it is not

         23        original to him is that you should verify the

         24        election, not the system, which I always rather

         25        liked, in the sense that is the spirit of these



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                207





          1        end-to-end systems.

          2             But of course, also echoing the fact that

          3        you want a fair degree of confidence that the

          4        election is going to run smoothly and correctly

          5        before you launch into it.  But you don't want

          6        to have to rely on the evaluation of code and

          7        so forth to have the assurance of accuracy in

          8        the end.

          9             So, again, Ron sort of commented that it

         10        has to be perceived to be accurate as well as

         11        vital, so I like that catch phrase.  I think

         12        that's all I wanted to say.

         13             >> KING:  Thank you, Peter.

         14             Ron?

         15             >> RIVEST:  Thank you.  Thanks for giving

         16        me the last word here.

         17             I would like to echo, thanks to the

         18        organizers for inviting me here.  It is a

         19        pleasure and a privilege to participate, and it

         20        has been a great panel, actually.

         21             I think this process is, actually -- the

         22        E.A.C. and everybody involved in putting this

         23        process together deserves to be commended.  It

         24        really is, I think serving the country well and

         25        getting the right issues on the table and



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                208





          1        getting debated.

          2             And having the draft VVSG out for public

          3        comment like this, I think, is exactly the

          4        right thing and the comments that are being

          5        debated here are just right on target.

          6             These issues aren't simple, and they

          7        deserve everybody putting their heads together

          8        to figure out what is the right approach.

          9             The VVSG, I just wanted to emphasize the

         10        three things that are the highlights for me of

         11        the new proposal.  Software independence, of

         12        course, is one, and I think that deserves a lot

         13        of discussion.

         14             I think it is the right way to go.  If you

         15        have concerns about it, I think the right

         16        question -- I would encourage you to ask -- if

         17        you would like to have systems that are outside

         18        of the software independence sphere certified

         19        somehow, how do you draft standards for them?

         20             That is one of the things that we really

         21        had the toughest part with trying to figure out

         22        in a situation where you've got a voting system

         23        where the count really depends on the software.

         24             How can you, in an economic way, try to

         25        get certification of such a software system out



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                209





          1        there?  And I think, really, we got stumped

          2        with that.  We couldn't do it, and I think

          3        software independence seems to be the cheaper,

          4        more trustworthy alternative, and trying to

          5        certify systems that depend so heavily on

          6        software seems to be just not something that

          7        this industry can reasonably support.  So S.I.

          8        is one.

          9             The OEVT is the second -- to me, a major

         10        interesting component.  I like John's proposal

         11        for renaming of that, as expert security

         12        review.  I think that is maybe a better name

         13        for that, maybe we can call it that.

         14             And the innovation class, I like new and

         15        better as well, a better mousetrap, or

         16        variations of class, I don't know.  Those are

         17        the three aspects, I think, I'll highlight.

         18             There have been a lot of issues brought up

         19        here, which brought to mind interesting echos

         20        of the whole debate within the TGDC, because

         21        there are a lot of things that are explicitly

         22        outside of what the scope of what the TGDC had

         23        to consider.

         24             The drafting of the VVSG, the proposed

         25        VVSG is merely an equipment certification



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                210





          1        standard.  It is not more than that, and there

          2        is so many important issues in voting systems

          3        that are outside of that that I think would

          4        have been wonderful if they could have been

          5        addressed in something like the VVSG, but they

          6        are just outside of this scope.

          7             Let me list some of those, and I think

          8        these are issues which I hope the E.A.C. will

          9        carry forward in some manner or the other to

         10        try to see that they get studied and addressed

         11        and approved somehow.

         12             The whole economic picture is one that has

         13        been brought up.  We have the funding of

         14        testing and so on, is certainly is one that I

         15        think would be nice to have worked up or a

         16        review of how that's done.  Are there

         17        innovative ideas in just sort of the funding

         18        structure there.

         19             The whole idea of procedures -- election

         20        procedures, that was outside of the scope of

         21        the VVSG, and they certainly impinge on

         22        security and so on.  I know the E.A.C. has

         23        already been active in that area, and that is

         24        great.

         25             The whole issue of standards for



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                211





          1        interoperability and component certification is

          2        an area which was outside of the scope of this

          3        VVSG, and I think it deserves thought, and it

          4        would be nice to see some movement forward on

          5        that.

          6             The issue of transition between current

          7        existing systems and new systems is one that I

          8        know the E.A.C. has been working hard at, and

          9        election officials are concerned about, as

         10        Sarah said, and I think that is also outside of

         11        the VVSG, per se.

         12             And the other issue that Alec raised

         13        earlier about the whole software development

         14        process is one that, again, is outside of what

         15        the VVSG could handle and was scoped to handle,

         16        chartered to handle.  And I think it was

         17        another gray area for work.

         18             So there is lots of ways for improving

         19        voting that is certainly outside of what the

         20        VVSG can attempt to do.  And I think as we look

         21        at the VVSG and send in comments and I think it

         22        is important to keep in mind what the scope of

         23        the VVSG is supposed to be and what is outside

         24        of the scope.

         25             And for things that are outside of the



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                212





          1        scope, don't drop those issues, but they are

          2        probably going to be reflected in a revised

          3        VVSG.

          4             That's all.  Thanks.

          5             >> KING:  Good.  Thank you, Ron.  I am

          6        about to end on time, which is a phenomenon in

          7        and of itself.  The first thing that I want to

          8        say is thank you to everybody on the panel

          9        today.  This has been an extraordinary

         10        classroom.  I think I have learned more today.

         11        I heard some old voices more clearly, and I

         12        heard some new voices who brought some new

         13        things to the table and this was especially

         14        satisfying for me.

         15             I did write down many things but in

         16        reflecting on how long the VVSG should be,

         17        whether 600 pages is enough to explain

         18        something that complicated or two pages, but I

         19        like a three-line voting standard that says

         20        cast has intended, collect as cast, and count

         21        as collected.  And I don't think I ever heard a

         22        more succinct expression of the business that

         23        we are about and I thank you for that, Ron.

         24             In my closing remarks, I hope that all of

         25        us keep in mind that the work of everybody on



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                213





          1        this project, the NIST personnel, the TGDC, all

          2        of the academicians who have made this now a

          3        part of the research is all critical, but it

          4        all funnels down to a relatively small number

          5        of people, the E.A.C., itself and the Standards

          6        Board who must become accountable for this

          7        document.  And I think we need to keep in mind

          8        the stress that that places on those

          9        individuals to look at something this

         10        complicated and make sense and look at the

         11        future of what they are doing.  So we wish you

         12        the very best in your meetings tomorrow and to

         13        the E.A.C. commissioners, we wish you the best

         14        as you work through this process.  I would like

         15        to ask everybody if they would to thank the

         16        panelists for their work here today.

         17             (Applause)

         18             >> KING:  Some people came very long

         19        distances.  Peter in particular.  Many of us

         20        dropped what we were doing at the end of our

         21        semesters to get here.  And so I do thank

         22        everybody for that.  And Brian, with that, I

         23        will turn it over to you.

         24             >> HANCOCK:  Thank you, Merle.  Let me

         25        just reiterate the thanks that you just echoed



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                214





          1        to the panelists on behalf of our E.A.C.

          2        commissioners and at least three of the four

          3        are here in the audience and our executive

          4        director.  I really do thank you for coming.

          5        We appreciate your openness and candor, your

          6        willingness to be here.  You know, we need you.

          7        We will continue to need you.  And I hope that

          8        we can feel free to call on all of you as we go

          9        down this road to the final adoption of this

         10        document.  Thank you once again.

         11             >> KING:  With that, this meeting is

         12        adjourned.  Thank you.

         13             Meeting adjourned at 6:00 p.m.)

         14

         15

         16

         17

         18

         19

         20

         21

         22

         23

         24

         25



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210
                                                                215





          1   STATE OF TEXAS
              COUNTY OF TRAVIS
          2
                             REPORTERS' CERTIFICATE
          3       UNITED STATES ELECTION ASSISTANCE COMMISSION
                             STANDARDS BOARD MEETING
          4                     DECEMBER 11, 2007

          5             We, the undersigned Certified Shorthand

          6        Reporters in and for the State of Texas,

          7        certify that the above and foregoing

          8        proceedings were taken down, as stated in the

          9        caption, and reduced to typewriting under ou

         10        direction, and that the foregoing represent a

         11        true, correct, and complete transcript of the

         12        said proceedings.

         13             SUBSCRIBED AND SWORN TO under our hand and

         14        seal of office on this the 23rd day of

         15        December, 2007.

         16                  ________________________________
                             April C. Balcombe, CSR, CRR, CCP
         17                  Texas CSR 5752
                             Expiration:  12/31/09
         18                  TEXAS CLOSED CAPTIONING
                             310 East 34th Street
         19                  Austin, Texas  78705
                             1-512-480-0210
         20

         21
                             _______________________________
         22                  Michelle Robertson, CSR, RPR
                             Texas CSR 4268
         23                  Expiration:  12/31/09
                             TEXAS CLOSED CAPTIONING
         24                  310 East 34th Street
                             Austin, Texas  78705
         25                  1-512-480-0210



                               TEXAS CLOSED CAPTIONING
                     310 East 34th Street, Austin, Texas  78705
                                    (512)480-0210