GitHub repository: FreeAndFair/ColoradoRLA

Issues:
issue_926: state: open
title: Update with pointers to related repos and progress since 2018
body: The README should point to up-to-date information on ColoradoRLA:

- The second phase of development of ColoradoRLA was performed by [Democracy Works](https://www.democracy.works/) between 
2018-06-19 and 2019-11-10. Their work is available at https://github.com/democracyworks/ColoradoRLA.
- Subsequently, the Colorado Department of State took over maintenance, and their latest version as of 2022 is at https://github.com/cdos-rla/colorado-rla. That repository should be rebased on top of the one at DemocracyWorks so it has the full version history.

It should also note that the version here is out-of-date and depends on some out-of-date versions of other software which could introduce security vulnerabilities.
user: nealmcb
issue_925: state: open
title: Replace "Colorado", "state" and "county with more generic labels
body: The "state dashboard" is really the dashboard for the audit coordinator. The "county dashboard" is really the dashboard for the entity in charge of manual examination of ballots. One county has expressed interest in allowing the public to watch one or both dashboards, for which it would be nice to strip all the Colorado-specific naming.
user: sfsinger19103
issue_924: state: open
title: User options for number in first round
body: Different users might have different philosophies about how many ballot cards to examine in the first round. Current software was designed to make the number as small as possible, per Colorado's request. Other users might choose to examine more ballots in the first round. It would be nice to have some options for the user.
user: sfsinger19103
issue_916: state: open
title: Freeze / Infinite loop in ComparisonAuditController.java
body: When a contest has a discrepancy, and requires a second round of auditing, and the random selection has multiple duplicates, it is possible for the system to enter an infinite loop during the audit.

This would happen at the end of the second round, because the system thinks, incorrectly, that there is still more auditing to do. It causes the web interface to be unresponsive, and freeze, while the county is waiting for an (unnecessary) third round to launch. The loop would be in ComparisonAuditController.java at `while (sorted_deduplicated_new_cvrs.isEmpty()) {`

https://github.com/FreeAndFair/ColoradoRLA/blob/430c0d7bd73f9f76522c7d061d535fc41707c559/server/eclipse-project/src/main/java/us/freeandfair/corla/controller/ComparisonAuditController.java#L541-L564

The fix is to delete the line 

            cvrai.setCounted(cvrai.counted() + multiplicity);
in `ComparisonAuditController.java` as shown in https://github.com/democracyworks/ColoradoRLA/pull/17/commits/ec87ea6b69fecee549a4ec4fc688cea7cc638856

See a more detailed explanation, and the fix in context, in the DemocracyWorks fork: https://github.com/democracyworks/ColoradoRLA/pull/17
user: nealmcb
issue_913: state: open
title: Ballot-polling workarounds
body: Add some python scripts (`parse_hart.py` and `analyze_rounds.py`) to allow ColoradoRLA to be used to record audit board ballot interpretations in a ballot-polling audit.
See tools/rla_utils/README.md for more information.

user: nealmcb
issue_911: state: open
title: way too many significant digits displayed for risk limit
body: After I entered a risk limit of 0.94%, the system reported the risk limit was 0.9400000000000001%
<img width="1166" alt="weird_risk_limit" src="https://user-images.githubusercontent.com/14353822/40322210-78b58b00-5ce6-11e8-8680-85b3b53c6a9c.png">

user: sfsinger19103
issue_910: state: open
title: Home page with traffic / interest statistics
body: Visitor statistics for this project's github codebase are available at https://github.com/FreeAndFair/ColoradoRLA/graphs/traffic
but it appears that only 14 days of data is currently available, unfortunately:
 https://github.com/isaacs/github/issues/399

A better way to gauge and track interest in the project would be useful, perhaps via a separate home page.
user: nealmcb
issue_909: state: open
title: Super-majority contests
body: Some contests require a supermajority for victory. For example [parcel tax ballot questions in California](https://ballotpedia.org/Parcel_tax_elections_in_California) pass only if 2/3 or more of the voters vote "yes". 

It would be nice if the software would support RLAs of such contests.
user: sfsinger19103
issue_908: state: open
title: Be more explicit in LICENSE.md about exceptions to AGPLv3
body: Chris Jerdonek writes: I think this language and approach could be improved:

"The code, except as explicitly noted, is licensed under the GNU Affero General Public License Version 3 (AGPLv3). A copy of the AGPLv3 is provided in the file agplv3.txt."

The reason is that this puts the burden on the user to look for the "explicit notes," which a priori could be located anywhere in the repository. So it's not so friendly for people wanting to use the code and knowing what they're getting. It would be better if the file could say what in the repo is and isn't licensed under the various licenses that the repo might contain, without having to search the code base.
user: sfsinger19103
issue_905: state: open
title: More efficient sampling and auditing of small contests and multi-card elections via SCORE data
body: The 2017 implementation always samples from the entire set of ballot cards across a county, and thus uses the county-wide count of ballot cards as the denominator in the calculation of the diluted margin for each contest in a given county. For a contest which only appears on a third of the ballot cards, this results in a diluted margin one-third the size, and requires three times as many ballot cards to be audited.

This is because we can't trust the CVRs, and the current ballot manifest doesn't contain independent evidence about how many voters voted in each contest.

Colorado's SCORE voter registration database records each voter who votes in an election, and what districts they are eligible to vote in, and can be used to derive an independent count of the number of voted ballot cards that should appear, for each county, in the CVRs.  So SCORE is the most likely source of useful evidence which can be used to support sampling from only CVRs which report votes in a contest of interest.

Note that using SCORE is problematic because data in SCORE is updated as voter registration changes come in from a variety of sources, and it is hard to get the data from SCORE that corresponds to the generation of each ballot. Contest-ballot counts from the CVRs should be appropriately reconciled with the SCORE data and used to establish upper bounds on the number of ballot cards containing each contest, and appropriate techniques with phantom CVRs etc. should be used when they don't match.

There has been discussion of procedures which could provide contest-ballot counts per-batch, but here we assume the counts are just per-county, per-contest.

An audit of the SCORE data should also be done, along with chain-of-custody audits etc.
user: nealmcb
issue_904: state: open
title: Graceful recovery from issues with CVR/BM files after launch of audit
body: If a county misses the upload deadline, or uploads files that are found (after launch of audit by State user) to be incorrect, it would be valuable to allow some kind of recovery in the software. (Is this possible while maintaining the integrity of the audit?
user: sfsinger19103
issue_903: state: open
title: Prevention of and/or graceful recovery from mistakes choosing contests to audit
body: There is a risk of human error in the choices of contests to audit. Mitigations might include:

- putting more information on the "Select Contests to Audit" screen (such as ballots cast, diluted margin and anything else that would help user choose correctly)
- creating a review screen for user (like the screen shown to Audit Board before finalizing interpretations of a single ballot card)
- giving State user an option to reset the list of contests to audit (only during an appropriate time window)
user: sfsinger19103
issue_902: state: open
title: Allow County to recover from discrepancies due to wrong ballot-pulling process
body: In 2017 one county Audit Board recorded many discrepancies because the physical ballot cards pulled to audit were not the correct ones. There was no way, using the current version of the software in this repo, to allow them to correct the process mistake and do the auditing over. 

It would be valuable to have a feature in the software allowing a County in a similar situation to start the audit board work over, canceling the false discrepancies.
user: sfsinger19103
issue_901: state: open
title: Support ballot-polling audits
body: Related to #900
user: nealmcb
issue_900: state: open
title: Support auditing multi-county contests
body:  State-wide contests and many federal and district contests include ballots from multiple counties.

We need better data to audit these contests, in which the CVR either guarantees that contest names are unique across counties, or provides some other sort of unique identifier.

Various options exist for the risk-limiting calculations needed for these contests.  [A Bayesian Method for Auditing Elections](https://people.csail.mit.edu/rivest/pubs/RS12y.pdf) by Rivest and Shen can also handle audits in which some counties are using ballot-polling and others are doing ballot-comparison audits.  See e.g. https://github.com/ron-rivest/audit-lab.  Other frequentist methods by Stark, at least for the case where all the counties use the same sort of auditing, are available, and more efficient ones are in development.

Given `rla_export` data from the 2017 election, we can evaluate the efficiency of the various approaches on real-world scenarios and decide how to move forward.
user: nealmcb
issue_898: state: open
title: rla_export: Fix license: GPL, via SPDX; note MIT for Dotable()
user: nealmcb
issue_896: state: open
title: Ballot Manifest Upload During CVR Import Fix
body: There was a missing transition in the state machine: namely, the transition from `CVRS_IMPORTING` (and also `BALLOT_MANIFEST_OK_AND_CVRS_IMPORTING`) to `BALLOT_MANIFEST_OK_AND_CVRS_IMPORTING` when a ballot manifest is uploaded while a CVR is importing asynchronously. This PR adds that transition.

It also undoes the relaxation of the `comment` field name on ACVR uploads that I had put in because I had thought the client bug related to comments was related to field naming rather than just not including the comments in the submission.
user: dmzimmerman
issue_880: state: open
title: Handle duplicate contest names gracefully
body: The current system pulls contest names from a CVR and can't distinguish two contests in a single county with identical names but different contest ids. Contest ids are not part of the CVR export. If multiple contests have the same name (ignoring any trailing white space) but accept different numbers of votes, the CVR file is reported as invalid.

Using ENR exports received to date based on LAT testing with the 2017 election definitions for Colorado, we have detected duplicates in a few counties, because the ENR exports include not just contest names but also contest ids (unique for each contest within the county).

For Nov 2017, we will use a workaround, to edit the CVR files, and ensure that contest names in CVRs used for audit will be unique within counties.
user: sfsinger19103
issue_876: state: open
title: --check-selection to verify random sequence etc.
body: Fix call to sampler.generate_outputs() in publish_ballots_to_audit
Finish get_county_cvrs()

Also:
For each audited cvr, show imprinted_id, not db_id

Exit on upload failure reasons, explain why
Fix bug in get_county_dashboard() acvr default

Streamline logging of upload progress
Switch from 30 second interval for progress uploads to 3
New arg: '--progress-delay', Time to pause between upload progress queries. Default 3.0.
Fix some broken cvr and acvr download examples
user: nealmcb
issue_874: state: open
title: update example Apache configuration for SOS breadcrumb
body: Update the Apache configuration so that `/sos/county` serves `index.html`.

user: kiniry
issue_866: state: open
title: Log off / exit button does not work in Safari
body: Safari Version 11.0 (12604.1.38.1.7), running on a MacBook Pro (macOS Sierra, v 10.12.6), logged in as a County

Exit button worked as expected in Firefox v 56.0.2 (64-bit) (logged in as a County)
user: sfsinger19103
issue_862: state: open
title: Specify minimum set of tables necessary for export for public Audit Center
body: Here's a minimum protocol which pretty much determines the minimum set of tables.

source: https://docs.google.com/document/d/1olo2W38kMi4mUH1S3UxRfFMlrszXHjLNK_dLuAMLT1w/edit#heading=h.ra35frr05gsz

Minimum protocol to check correctness of Colorado’s Risk-Limiting Tabulation Audit
If an independent entity wishes to check the correctness of the Colorado risk-limiting tabulation audit, it is sufficient for that entity to carry out the following steps. We believe these to be a minimum set of necessary items to verify that the audit was correctly conducted.

Note that this protocol does not address certain aspects of public verification of the election results, but only the correct performance of the Risk Limiting Audit from the ballot cards in custody of the Counties and the ballot manifest and CVR files exported from the voting system. In particular, chain of custody issues for the ballot cards before tabulation are not addressed by this protocol.

There is no break in the chain of ballot card custody between the tabulation of the results by the system under audit and the Audit Board review of ballot cards.
Observation: details of observation depend on chain of custody protocol.
Tabulating contests “selected for audit” on the CVRs used for the audit yield the announced tabulated results. Requires:
Data: CVR files
Data: List of contests selected by Secretary of State for audit
Data: Announced tabulated results for contests selected for audit
Calculation: tabulate results in contests selected for audit from CVR entries, compare to announced results. These should match.
The set of ballot cards listed in each ballot manifest exactly matches the set of ballot cards represented in the corresponding CVR file. Requires:
Data: The ballot manifest files
Data: The CVR files
Calculation: match the data lines in the CVR file one-to-one with the cards listed in the corresponding ballot manifest file
The list of ballot cards selected for audit has been determined after the finalization of the manifests and CVR files. Requires:
Data: Hashes of CVR and ballot manifest files (released to public before selection of random seed)
Data: full CVR and ballot manifest files (can be released later)
Calculation: calculate hashes of the full files and compare to the hashes released before the selection of the random seed. These must match.
Random sequence of ballot cards has been properly selected according to the specified random algorithm. Requires:
Observation: Random seed generated at public meeting
Data: Random sequence of ballot cards used for the audit. (This random sequence may include duplicates.)
Data: Ballot manifest file
Data: CVR file
Calculation: Use ballot manifest file to number ballot cards represented in the ballot manifest sequentially. Then calculate the pseudo-random sequence via prescribed algorithm based on the random seed. Together these determine the random sequence of ballot cards. This should match the sequence used in the RLA. 
The list of ballots selected for audit matches the list actually reviewed by the Audit Board.
Data: CVR file
Calculation: match lines of the CVR file to the ballots listed in the ballot manifest. 
Data: List of ballot cards assigned for review by Audit Board. This list can be created from the random sequence by removing duplicates.
Observation: physical assembly of ballot cards for Audit Board review
Audit Board interpretation matches content of paper ballot cards
Observation: Audit Board review of paper ballot cards
Audit Board intent is reflected in the computer record of Audit Board intent.
Observation: Audit Board review of ballot cards
Data: computer record of Audit Board review
Calculation: the records from the observations should match the computer record.
Risk limit achieved (or hand count ordered)
Data: status of each contest selected for audit -- in particular, which contests if any has the Secretary of State selected for hand count? 
Data: the “risk limit” for each contest selected for audit.
Data: the “error inflation factor” used to calculate whether the risk limit has been achieved
Data: the “diluted margin” for each contest under audit, i.e., the vote difference between the lowest winning total and the highest losing total, divided by the total number of ballot cards in the county. This can be calculated from the CVR file.
Data: for each contest under audit, the number of each type of discrepancy found in that contest by comparing Audit Board interpretations to the original CVR file. These can be calculated by comparing the entries in the CVR file to the corresponding entries in the record of Audit Board interpretations. The types of discrepancies are:
Two-vote overstatements (Audit Board finds a vote for a loser where original CVR showed a vote for the winner)
One-vote overstatements
One-vote understatements
Two-vote understatements
Calculation: Using the formula for the stopping size given on Stark’s Audit Tools web page, see if the size of the sample checked so far exceeds the stopping sample size. With the following notations
a	risk limit
g	error inflation factor
m	diluted margin
o2	number of two-vote overstatements
o1	number of one-vote overstatements
u1	number of two-vote understatements
u1	number of two-vote understatements

the formula for the stopping sample size is:

	stopping sample size = -2g(log(a) + o1log(1-1/(2g)) + o2log(1 - 1/g) + u1log(1+1/(2g)) + u2log(1+1/g)) / m)

user: sfsinger19103
issue_860: state: open
title: fully document default.properties
body: We need to fully document the `default.properties` file.
user: kiniry
issue_855: state: open
title: precise, automated timing and rate analysis for load test scripts
body: Evolve our current primitive load test scripts so that precise and automated timing and rate analysis is emitted for various configurations, preferably into a CSV file for post-analysis in a spreadsheet.
user: kiniry
issue_854: state: open
title: Load tests
body: Closes #350 , #351.
user: kiniry
issue_844: state: open
title: Doc tweaks for User Guide generation
user: kiniry
issue_797: state: open
title: Simultaneous recall elections
body: 19.  The counties can upload files to the RLA tool before the state defines the election. That works if only one election is being conducted at a time. However, in the case of 2 recall elections being conducted on the same date how does the state conduct two separate audits? How do the counties indicate which election the updated files belong to? Should the counties be able to upload files before an election is defined?
user: sfsinger19103
issue_776: state: open
title: Add owasp dependency-check-maven plugin for server security report
body: Run it via `mvn dependency-check:check`
user: nealmcb
issue_775: state: open
title: Spinner doesn't go away until you log off for 33K or more CVRs (in Chrome)
body: Verbal description from CDOS: at 33K CVRs the spinner doesn’t go away until you log off, or until the state starts the audit.

[We expect to receive a more carefully documented description, with screenshots, by the end of the week.]
user: sfsinger19103
issue_772: state: open
title: County file download buttons
body: Users are now able to download ballot manifest and CVR export files for (1) their own county (if a county user) or (2) all counties (if they are a DOS user).

- The download buttons for the county dashboard is on the county home page.
- The download buttons for each county for the DOS user is in the county _detail_ page, e.g. `/sos/county/1` for Adams county.

The UI for this was implemented in #771, and now needs to be documented.
user: ranweiler
issue_761: state: open
title: If handcount is declared, what should show in SoS County Update Table?
body: Here's what we have (after a hand-count was declared for Adams County, the only county in this particular test audit. What should be in the "Est. Remaining Ballots" column?

<img width="1198" alt="handcount" src="https://user-images.githubusercontent.com/14353822/30725058-d0f3a076-9ef6-11e7-9b9c-5e8919109a90.png">

user: sfsinger19103
issue_750: state: open
title: Votes received during the audit in County Audit Report?
body: (per RLA Tool Feedback #1_tool version2)

Suggestion (not requirement) from CDOS:
> [In County Audit Report] it might be useful to know how many votes each candidate received during the audit, and how many were discrepancies or not.

It's not clear to me what exactly the suggestion is: is it the candidate (audited) vote totals from audited ballot cards alone? Or total votes, "correcting" any discrepancies found during the audit?  My guess is, for each candidate three columns:

- Corrected vote total (original vote total plus net votes corrected by audit) 
- Number of cards whose audit increased the candidate's vote total
- Number of cards whose audit decreased the candidate's vote total.

If I understand correctly, this would require tracking things we don't currently track (discrepancy per candidate, not just per contest).


user: sfsinger19103
issue_748: state: open
title: Election date off by one in County Audit Report
body: (per RLA Tool Feedback #1_tool version2)

CDOS reports date of 8/21 in report even though 8/22 was entered as date of election.
user: sfsinger19103
issue_738: state: open
title: setup release signing infrastructure
body: - [ ] create Free & Fair key pair and register it in the appropriate places
- [ ] create a Free & Fair Keybase account and set it up appropriately
- [ ] sign past and future releases in an appropriate fashion using these keys
user: kiniry
issue_736: state: open
title: Example properties for proprietary 2FA.
body: This PR simply includes a few new lines to show how to point to a different (possibly proprietary) implementation of our 2FA API.
user: kiniry
issue_732: state: open
title: Haphazard markdown formatting in markdowns for end user docs
body: Because I'm not familiar with the effects of various commands and conventions in .md files, I haven't been using (or removing) any such commands that don't show up in the docx versions of the documents. This situation should probably be repaired, possibly by me with support from someone who has experience with md, pandoc and the various other types of files produced (LaTeX? straight to pdf? html? I don't even know the list!). Or possibly by someone who can just go in and do the cleanup.
user: sfsinger19103
issue_695: state: open
title: Robust estimation of ballots for next round
body: There are some corner cases of unusual inputs, encountered during testing, in which the calculations from the `nminFromRates()` function in https://www.stat.berkeley.edu/~stark/Vote/auditTools.htm are invalid. An interim robust-but-inaccurate algorithm which leverages the robust and accurate results from the `nmin()` function is in use now for estimating ballots-to-audit for future rounds.

Replace this interim algorithm with a better one, which takes into account the currently observed discrepancy rate.

See also #422 , #482
user: nealmcb
issue_690: state: open
title: Add third option to list of reasons to audit.
body: From CDOS RLA Tool Feedback #2
9/1117

At present the tool does not support auditing the same contest across multiple counties so the selection in not critical. A third option should be added in the future to be able to audit multi-county-jurisdication contests that are not state contests.

user: sfsinger19103
issue_680: state: open
title: Use flags for cookies
body: On the server, enable `Secure` and `SameSite=Strict` in the `Set-Cookie` header when in production mode (only). In all cases, enable `HttpOnly` for the session cookie and any other cookie that does not need to be accessible to scripts.

- [x] `HttpOnly`
- [ ] `Secure`
- [ ] `SameSite=Strict`
user: ranweiler
issue_671: state: open
title: Print a tracking form for each ballot card selected for audit
body: It would be helpful if the audit tool could print a tracking form for each ballot card selected for audit.
The form could include:

* Ballot location (e.g. Scanner #, Batch #, Ballot Position #, Storage bin)
* BallotType / Ballot Style
* A bar code with that same information, making it easy for mobile apps, or plain cameras, to take a picture of the tracking form and the ballot, for later publication, analysis etc.
A scanner wand attached to the audit tool entry terminal could also recognize the bar code and automatically pop up the audit board ballot card interpretation form to fill in
* Round number
* Sequence number in round by location
* Sequence number in the overall random selection
* Timestamp of printing time
* Space for notes and to optionally enter other tracking information:
    * Timestamps of when the team went to retrieve the ballot and when it was located
    * Discrepancies
    * Investigation status

Some day some of this information could also be tracked via a simple smart phone or tablet app

user: nealmcb
issue_640: state: open
title: Formatting disappears on zoom-out in Safari v10.1.2
body: Before zooming out:
![safari_before_zoomout](https://user-images.githubusercontent.com/14353822/29998574-a22dd210-8fe3-11e7-97d8-8ef2d1084113.png)

After zooming out:
![safari_after_zoomout](https://user-images.githubusercontent.com/14353822/29998575-a2382382-8fe3-11e7-9828-20a6712ebaf1.png)

user: sfsinger19103
issue_614: state: open
title: use columns to render ballot verification page 
body: On many displays we can fit two columns, if not three, to avoid vertical scrolling.
user: kiniry
issue_604: state: open
title: Tooltips and pop-up help messages on all UI elements
body: All non-trivial UI components should have pop-up help.  E.g., all column labels in Dashboards, the logout button, etc.
user: kiniry
issue_603: state: open
title: ensure that the precise format of our logs matches the CDOS specification
body: Here is the relevant fragment of the RFP's requirements:

> At a minimum the following entries for all system components must be logged.
> - User ID
> - Type of Event
> - Data and Time
> - Success for Failure indication
> - Origination of event
> - Identity of affected data or system

user: kiniry
issue_599: state: open
title: Ability to wrap up a round early
body: It would be helpful if the tool could notice that a county has met all its risk limits early, even though all the ballots for the pre-specified round size have not been entered.  This may happen, e.g. if there are some discrepancies resulting in understatements of the margin.  If this happens, the tool could offer to let them submit their final report and be done.
user: nealmcb
issue_598: state: open
title: Option to continue auditing, gathering more evidence
body: Public interest may develop in a contest which is not selected for an RLA and thus driving the audit. E.g. some discrepancies may be noted, or there may be other reasons that develop after official RLA contest selections are made.

If a county wants to continue gathering evidence in order to meet a risk limit for such an opportunistically audited contest, it should be able to specify and obtain an additional set of ballots to audit from the sample, e.g. based on their own risk level calculations.

user: nealmcb
issue_597: state: open
title: Noting contests not present on the Ballot Interpretation entry page
body: One form of malfunction or attack on an election involves votes in contests on a ballot that just don't show up in the associated CVR.

If the audit board sees a contest on the ballot that the tool doesn’t display (because it is not in the CVR), the tool should allow them to indicate this. Currently this kind of problem could simply be noted as text under "Comments" for one of the existing contests, but having a checkbox for "contest not found", or an overall comments form for the whole ballot card might be clearer and easier to track. The recommended procedure should be documented.
user: nealmcb
issue_596: state: open
title: Support for untabulated ballots: provisionals etc.
body: Audits ideally come after all the votes are tabulated, canvassed and reconciled. In Colorado, however, the certification deadline comes shortly after the last date for voters to cure signature verification problems, for provisional ballots to be processed, etc., It is thus possible that a very small number of such ballots may be tabulated after the deadline for CVRs and manifests to be uploaded to the ColoradoRLA tool.  An updated vote count including such ballots may be released at the end of the tabulation and canvass, and the risk limit of the audit needs to apply to the updated outcome.

This is allowed e.g. by Colorado's Rule 25, e.g. for provisional ballots.

RLA theory covers this case. As discussed in [Banuelos-Stark-2012](https://arxiv.org/abs/1207.3413), counties should add a batch of _phantom_ ballot cards to the manifest, which do not correspond to any CVRs, one for each possible late-tabulation ballot card.

To support this, if the total number of ballot cards identified in the manifest is greater than the number of CVRs, the server should augment the sample space for the random selection from the CVRs to match the total from the manifest.  Any selections that don't map to an uploaded CVR should be automatically treated as phantom ballot cards, and handled as in the existing "ballot-not-found" case, i.e. as if votes were marked for all the losers.
user: nealmcb
issue_595: state: open
title: Clarify that all contests are subject to audit
body: ColoradoRLA lets the Audit Center put contests in two categories.

* It selects some contests for mandatory risk-limiting audit, and these "drive" the audit, determining how much sampling and interpretation is needed to establish that they have all met their risk limits, and thus when the audit is finished.

* The other contests are still audited to some degree, since the Audit Board enters vote interpretations for all contests on the sampled ballots, which can be analyzed to see whether risk limits have been attained even for those contests that have not been selected for mandatory risk limiting audit. When risk limits aren't met, risk levels could be measured.

Some of the text in the UI lists the second category of contests as "not selected for audit". It would seem more accurate to label them as "opportunistic audit", or with some other description that emphasizes that it is being audited, but just not _driving_ the audit.
user: nealmcb
issue_579: state: open
title: add second factor card # to auth protocol
body: So that we can mimic the SCORE login as closely as possible.
user: kiniry
issue_528: state: open
title: System must work in Internet Explorer
body: from 8/28 CDOS feedback: 

"It does not work in Internet Explorer." Presumably "It" is the RLA Tool.
user: sfsinger19103
issue_496: state: open
title: Refactor ASM-based redirects
body: If a user should be unable to access a client-side URL path because of the current ASM state, this logic should all exist in one place, somewhere near the `RootContainer`. As it is, we have it scattered amongst the few top-level page containers that enforce such constraints.
user: ranweiler
issue_488: state: open
title: Verify semantics of the CVR upload, and that CVR outcomes match the preliminary reported outcomes from Election Night Reporting and/or the summary report
body: In order for a county to verify that a CVR was properly uploaded, parsed and tabulated, and that it represents the same view of the election as the other exports of the system, more details on the tool's interpretation of the data should be shared after upload. This includes the number of CVRs, and, for each contest, the number of CVR cards with that contest on them, the contest winners, and the tightest margin of victory (the winner with the least votes minus the loser with the most votes). Providing initial sample sizes would also be helpful.

After #477, the response from `/import-cvr-export` after a cvr upload looks like this:
```
<Response [200]> /upload-file {
  "file_id": 53375,
  "county_id": 3,
  "filename": "arapahoe-regent-3-clear-CVR_Export.csv",
  "size": 10968,
  "timestamp": "2017-08-26T02:32:35.179Z",
  "hash_status": "VERIFIED",
  "status": "IMPORTED_AS_CVR_EXPORT"
}
```
Augmenting it to add just the CVR count would be a significant step to ensure that the right data was uploaded.

Additional steps to allow counties to ensure that the contest winners and margins match should also be taken, either within the tool's web user interface, or via a csv download of contest results, or via some out-of-band method with an appropriate user interface.
user: nealmcb
issue_481: state: open
title: Add script to sort glossary entries.
body: As discussed [here](https://github.com/FreeAndFair/ColoradoRLA/issues/331#issuecomment-324986921).

I don't know where you wanted the script, so I just put it at the top level. Also, I noticed there's a redundant file `docs/glossary_concepts.txt`, so I removed it.

user: cjerdonek
issue_472: state: open
title: how to handle fictitious CVRs?
body: @nealmcb attempted to create a bogus CVR file by duplicating the same CVR line many times.  How might the RLA Tool detect such bogus CVR files?  CC @dmzimmerman 
user: kiniry
issue_422: state: open
title: estimated error rates for stage/round one RLA batch size
body: Stark's calculator uses estimated error rates for:
 - one vote overstatements of 0.001 (0.1%, or 1 in a thousand audited ballots contain such an error), 
 - two vote overstatements of 0.0001 (0.01%)
 - one vote understatements of 0.001 (0.1%)
 - two vote understatements of 0.0001 (0.01%)

@nealmcb suggested initially that we use estimated error rates customized for Colorado of 0.01 for all four of these parameters.  Experimenting with such rates leads @dmzimmerman and I to conclude that this is likely too conservative, as the number of ballots necessary to audit is relatively large, albeit comparable to the volume of audits under Colorado's previous statutes (500 ballots or 5% of all ballots, whichever is smaller).

What we really need is information from either the EAC VVSG test reports for Dominion's Democracy Suite, or a direct conversation with Dominion, that tells us what kinds of error rates their equipment witnesses in field use.  If we had that data, we could/should use it in our calculations.

CC @dmzimmerman @nealmcb 


user: kiniry
issue_412: state: open
title: Investigate ClassNotFoundException in C3P0
body: Occasionally, when I leave the server running and idle for a while, I see one of these pop up in my terminal:

```
Exception in thread "C3P0PooledConnectionPoolManager[identityToken->1hge8tw9q3j483e1ooj1bn|40ef3420]-AdminTaskTimer" java.lang.NoClassDefFoundError: com/mchange/v2/resourcepool/BasicResourcePool$1DestroyResourceTask
	at com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1092)
	at com.mchange.v2.resourcepool.BasicResourcePool.removeResource(BasicResourcePool.java:1558)
	at com.mchange.v2.resourcepool.BasicResourcePool.removeResource(BasicResourcePool.java:1528)
	at com.mchange.v2.resourcepool.BasicResourcePool.cullExpired(BasicResourcePool.java:1616)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$1900(BasicResourcePool.java:44)
	at com.mchange.v2.resourcepool.BasicResourcePool$CullTask.run(BasicResourcePool.java:2149)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.ClassNotFoundException: com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 8 more
```

I attempted to address it in a previous PR, but it is unclear whether it is still occurring; it needs investigation.
user: dmzimmerman
issue_400: state: open
title: Ongoing specification of RLAs in general, and RLAs for Colorado.
body: In particular, I'll be focusing on their notion of rounds, or what are called stages in the literature.
user: kiniry
issue_393: state: open
title: Consider Logging Integrity Issues
body: Right now, the logs in the database are hashchained - you can pick any log entry, follow the chain of hashes back to the root, and know that no entry in between has been tampered with. However, it is possible, through the magic of optimistic locking and concurrent transactions, for the log to become a tree rather than a single linear sequence. In that scenario, one could potentially remove an entire branch of the tree without it being detectable in an investigation of the logs. 

Possible solutions for this include:
- implementing an actual Merkle tree, as compared to the upside down one we have now - computationally very costly with each transaction
- checking at every log commit to see if there are in fact multiple branches, and if so, adding a "merge" log entry to tie the branches together before committing new logs (not clear how well this would actually work, though, and it still gives a window, albeit very small, for log tampering)
- ???

user: dmzimmerman
issue_375: state: open
title: User should be able to change focus and act on forms with keyboard
body: E.g., pressing return or tab should move to the next textbox, etc.
user: kiniry
issue_357: state: open
title: include copies of all license files for dependencies
body: I like the Apache Foundation's common practice of having them all collected in a subdirectory at the root of the project.  

E.g., from their LDAP API:
```Free--Fair-2:apache-ldap-api-1.0.0-bin> pwd
/Users/kiniry/Sandboxes/LDAP/apache-ldap-api-1.0.0-bin
Free--Fair-2:apache-ldap-api-1.0.0-bin> ls licenses/
antlr-LICENSE.txt			xml-apis-LICENSE.dom-documentation.txt	xml-apis-README.sax.txt
dom4j-LICENSE.txt			xml-apis-LICENSE.dom-software.txt	xml-apis-README.txt
jbcrypt-LICENSE.txt			xml-apis-LICENSE.sax.txt		xpp3-LICENSE.txt
slf4j-LICENSE.txt			xml-apis-README.dom.txt```
user: kiniry
issue_351: state: open
title: design and implement scenario for testing DB performance in both single and multi-instance configurations
user: kiniry
issue_350: state: open
title: design and implement automated load testing of full system
user: kiniry
issue_349: state: open
title: integrate JSON-format Election Definition files into into our knowledge base and Q/A process
body: Neal sent an email to Denver 08-06 inquiring whether we can put the data they sent in our repo.
user: kiniry
issue_345: state: open
title: ensure transport subsystem is fully documented in specification
body: In particular, make sure all JSON and endpoint specifications reflect the actual implementation and the protocol specification is up-to-date.
user: kiniry
issue_344: state: open
title: ensure persistence subsystem is documented in book
body: In particular, make sure we include a description of all of the core design matters and subtleties of our use of Postgres (in a failover configuration) and our use of Hibernate.
user: kiniry
issue_343: state: open
title: complete transport subsystem
user: kiniry
issue_342: state: open
title: complete persistence subsystem
body: We have a working persistence subsystem for our `phase-1` delivery.  In order for us to call it complete it must:
- [X] be performance tuned for deployment
- [ ] support a failover distributed & parallel deployment
- [ ] be fully documented in the book and the formal specification
user: kiniry
issue_341: state: open
title: ensure persistence subsystem is fully documented in specification
body: In particular, every automatically generated table in the deployed DB should map to a `table` concept in the specification, and every query realized in the `us.freeandfair.corla.query` package should be realized by a function in the spec.
user: kiniry
issue_340: state: open
title: ensure transport subsystem is documented in book
body: Included in this documentation is our use of ASMs to describe client and server state, our specification of endpoints, our use of JSON and specification of transport types, and our protocols.  Much of this is already specified in our formal specification.
user: kiniry
issue_338: state: open
title: explain how new transport concepts are designed, implemented, and added to system
body: This is mainly for future developers outside of Free & Fair that wish to extend the system and easily understand its transport subsystem and our use of JSON and HTTP.
user: kiniry
issue_337: state: open
title: explain how new persistent concepts are designed, implemented, and added to system
body: This is mainly for future developers outside of Free & Fair that wish to extend the system and easily understand its persistence subsystem and our use of Postgres and Hibernate.
user: kiniry
issue_335: state: open
title: automatically enumerate all warning or error events
body: A nice cross-cutting way to enumerate all such events and ensure that the refine correctly across specification to client to server would be lovely.
user: kiniry
issue_332: state: open
title: finish top-level README.md
body: Ensure top-level documentation (README.md) summarizes project and all relevant subcomponents and their documentation.
user: kiniry
issue_331: state: open
title: ensure glossary is up-to-date, appropriate, and consistent for User Manual, Run Book, and Specification
user: kiniry
issue_328: state: open
title: render server ASMs
body: Per @dmzimmerman's work on #327.
user: kiniry
issue_289: state: open
title: Add requirements.txt for Python smoke tests.
body: Hi!

I was reading the README for the smoketest directory and noticed that you can use a `requirements.txt` file. That's a best practice that simplifies test setup and also ensures that test environments are deterministic / identical across users (by pinning dependencies).
user: cjerdonek
issue_278: state: open
title: Wrapper Method for AbstractEndpoint.endpoint()
body: Right now, every single endpoint() method ends with `return my_endpoint_result`. This seems ripe for refactoring, i.e., by having the overridable method be something like `endpointImpl(Request, Response)` and having the `endpoint()` method handle the return and also handle things like validating parameters (there is already another overridable method, `validateParameters(Request)`, with a default `skip` implementation, but we don't use it in this way). 
user: dmzimmerman
issue_275: state: open
title: Review deliverable vs Requirements and Design Decision Considerations
body: Review phase-1 deliverable vs _Requirements and Design Decision Considerations_ document.
File issues here as needed to refine the way the RLA algorithm interacts with the user interface, clarify where we're at and where we're going, and get CDOS input on relevant topics.
user: nealmcb
issue_261: state: open
title: First example auto-generated JSON schemas.
body: The web server https://jsonschema.net/#/editor was used to process
captured JSON samples from Neal's smoketest work.
user: kiniry
issue_254: state: open
title: In smoketest, integrate pytest sequences into main.py
body: In `test/smoketest`, the user currently needs to run `pytest server_test.py` separately from `main.py`. Integrate these, if it is possible to preserve the benefits of the pytest reporting and command line flexibility while running from the `main.py` code.
user: nealmcb
issue_252: state: open
title: turn DISABLE_AUTHENTICATION into a server property
body: See `AbstractEndpoint`.
user: kiniry
issue_249: state: open
title: implement queries to detect synchronization points between dashboards
body: According to the spec, we have two synchronization points that trigger internal state changes:
 - the DOS Dashboard should permit state administrators to choose which contests are under audit only when all state administrators have uploaded their verified CVRs correctly
 - the State Dashboard should permit county administrators to submit their final audit report only once auditors have finished their audit


user: kiniry
issue_246: state: open
title: Resolve IndicateHandCount/SelectContestsForAudit endpoints
body: Right now, these two endpoints have identical code: it takes an array of `ContestToAudit` objects as the request, and updates the DoS dashboard with them. This allows it to be used to:

1. select contests to audit
2. select contests for hand count
3. deselect contests for either of those things

Internally in the dashboard, all contests under audit or handcount are stored with the same kind of record (`ContestToAudit`), which identifies whether it is under audit or hand count. It seems like it's overcomplicating things to have two endpoints that do the same thing, but we might want them to actually do validation on their input (so that one _only_ takes contests to audit, and the other _only_ takes contests to hand count; if we do that, though, it's not clear how mistakes in input from the DoS side can be corrected - and in fact, to allow that to happen, we probably need a state machine change anyway).
user: dmzimmerman
issue_232: state: open
title: ensure all returned collections are immutable
body: Review the API and use `Collections.unmodifiableList` and similar methods or classes judiciously.  Right now we are inappropriately leaking private references willy-nilly.
user: kiniry
issue_229: state: open
title: Implement differing default risk limits for state-wide vs county-wide contests
body: Based on the adopted Colorado 2017 Rule 25, and a change in Rule 25 since the July 6 2017 draft, in the future the RLA tool will need to support multiple default risk limits.  It will also need a way to distinguish state-wide vs county-wide vs other contests.

> 25.2.2 PREPARING FOR THE AUDIT
>  (A) RISK LIMIT. NO LATER THAN 32 DAYS BEFORE ELECTION DAY, THE SECRETARY OF STATE WILL ESTABLISH AND PUBLISH ON THE AUDIT CENTER THE RISK LIMIT(S) THAT WILL APPLY IN RLAS FOR THAT ELECTION. THE SECRETARY OF STATE MAY ESTABLISH DIFFERENT RISK LIMITS FOR COMPARISON AUDITS AND BALLOT POLLING AUDITS, AND FOR AUDITS OF STATEWIDE AND COUNTYWIDE CONTESTS. IN COMPARISON AUDITS THE RISK LIMIT WILL NOT EXCEED FIVE PERCENT FOR STATEWIDE CONTESTS, AND TEN PERCENT FOR COUNTYWIDE CONTESTS.
user: nealmcb
issue_222: state: open
title: Ensure browser client state transitions match formal spec
user: ranweiler
issue_215: state: open
title: Allow configuration of gamma and discrepancy rates for RLA calculations
body: `src/main/java/us/freeandfair/corla/comparisonaudit/ComparisonAudit.java` defines a constant GAMMA = 1.03905, and allows the specification of the rate of one-vote and two-vote understatements and overstatements.
The defaults for these parameters are the kinds of numbers that users might want to tune in order to get appropriate initial sample sizes based on their expected rates, and (for gamma) the balance between expectations of one- and two-vote discrepancies.

E.g. some preliminary discussions suggest that setting gamma to a higher value like 1.1 (110%, as used in [_Super-Simple Simultaneous Single-Ballot Risk-Limiting Audits_, Stark 2010](https://www.usenix.org/legacy/events/evtwote10/tech/full_papers/Stark.pdf)) might be more appropriate if a significant number of two-vote discrepancies, due for instance to phantom ballots, might be seen.

A browser interface for specifying defaults for these is not in the current plan, but allowing defaults to be specified in the Java properties file (`src/main/resources/us/freeandfair/corla/default.properties`) seems appropriate.
user: nealmcb
issue_182: state: open
title: Improve user interface look and feel
body: The existing user interface is extremely minimal. Devise an implement minimal style guide for a nice look and feel with a consistent design language.
user: ranweiler
issue_123: state: open
title: Factor out file uploader as higher-order component
user: ranweiler
issue_68: state: open
title: Add round-trip tests
body: ~Blocked on #65.~

Test the behavior of the browser client against various XHR scenarios.
user: ranweiler
issue_55: state: open
title: Document how to verify Eclipse download
body: See discussion in #24.
user: ranweiler
issue_9: state: open
title: Add code coverage for client
user: ranweiler
issue_5: state: open
title: Minimize client bundle size
user: ranweiler