Administrative Office of the U.S. Courts 

CM/ECF

How to get started

building a new

CM/ECF. Today. 

A tactical approach to designing, building, and shipping products that solve real 

problems for real users in the federal Judiciary, immediately. 

Team 

Allison Press 

Ben Peterson 

Colin Murphy

Jessica Marine 

Matt Henry

Mark Meyer

Stacy Dion 

This guide builds directly on recommendations delivered by 18F in two previous 

reports: the Path Analysis Report and the Experiment and Iterate Report. We 

suggest reading those documents first. 
 

 

 

 

Table of Contents 

1. Executive Summary .......................................................................................... 3 

2. Research methodology .................................................................................... 4 

3. Establish an empowered Product Team, with a Product Owner ................. 6 

4. Conduct research with real users to direct all product decisions ............. 19 

5. Put existing data ownership and risk standards into practice ................... 24 

6. Start building the solution, incrementally ..................................................... 32 

7. Conclusion ....................................................................................................... 37 

8. Appendix A: Summary of recommendations ................................................ 38 

9. Appendix B: Unresolved Questions ............................................................... 39 

10. Appendix C: How to Proceed with COTS ..................................................... 42 

11. Appendix D: Example of a COTS Sandbox User Scorecard ....................... 45 

12. Appendix E: Example of an Overall Sandbox Evaluation Scorecard .......... 47 

2 
 

 

 

 

 

 

 

 

Executive Summary 

The reports from the Path Analysis and Experiment and Iterate phases of this project 

repeatedly showed how the siloing of the teams responsible for building CM/ECF has

resulted in failure of previous modernization efforts. 

“We talk about modernizing and the future of case management. Look at what we 

have given the courts over the last 20-30 years. We haven’t given them anything 

new, just regurgitated old stuff into new platforms. Before CM/ECF, AMES, ICMS -

we’ve only given them a new interface.” 

— AO stakeholder 

In addition to the recommendations outlined in the 18F Path Analysis report, we 

believe the Administrative Office of the U.S. Courts (the AO) should focus on 4 

core objectives right now which are most critical to the success of this initiative: 

1. Establish an empowered Product Team with a Product Owner

2. Conduct research with real users to direct all product decisions

3. Put existing standards for ownership of data and risk into practice

4. Start incrementally building the solution

Each office of the AO has made progress adopting agile methodologies individually,

but until the organization adopts agile practices across offices, silos will persist. So in 

addition to these 4 recommendations, it’s imperative that the AO implement these

recommendations outside of the existing structures, processes, and roles across

the Judiciary (the AO and courts). 

“When a company’s capabilities reside in its processes, and when new challenges 

require new processes—that is, when they require different people or groups in a 

company to interact differently and at a different pace than they habitually have 

done—managers need to pull the relevant people out of the existing organization 

and draw a new boundary around a new group.” 

Clayton M. Christensen and Michael Overdorf, March 2000 

The overarching goal is to create a product that meets user needs and a 

secondary goal is to create an organization capable of building and maintaining 

that product. 

3 
 

Research methodology 

Discovery 

The current 18F team drew on the valuable research conducted during the Path 

Analysis and Experiment and Iterate phases in order to build on the recommendations 

made in those reports. 

Our team’s primary research consisted of semi-structured interviews facilitated by a 

visual presentation or note taking. The benefit of this interview format was that it 

directed our conversation towards topical areas that built upon our initial research, 

while creating an open environment for follow-up questions and dialogue for richer 

understanding and insight. In addition to those interviews, we attended Enterprise 

DevSecOps Ad Hoc Working Group meetings and the regular Scrum1 ceremonies 

(meetings) of an CM/ECF agile development team to observe their discussions and 

identify organizational dependencies that came up. 

The goals of our research were to: 

● Understand the current state of the organization including product decisions,

development, and delivery at an executional level

● Present the strategic recommendations from our Path Analysis to relevant

stakeholders, get feedback, and discuss tactical considerations for

implementation

Design and Feedback 

As we began identifying tactical solutions, we sought inspiration from sources within 

and outside of the AO, including: 

● Interviews with 18F subject matter experts

● Voluntary interviews with senior agency stakeholders from other 18F

engagements to learn from the successes of other agencies

● Reviewing Executive Branch policies and guidance

● Reviewing public and private sector case studies

As we were developing our recommendations, we shared early rough drafts of our 

report with AO stakeholders in order to host semi-structured feedback sessions. These 

1 Scrum is a framework for agile product development centered around a self-organizing team, customer 

focus, and responding to change. 

4 
feedback sessions allowed us to share early concepts so that we could gauge their 

implications, assess areas of confusion, and refine them to meet the Judiciary’s needs. 

Participants 

● For discovery, we conducted 21 interviews and activities

○ With leaders and subject matter experts (SMEs) across the AO

● For design and feedback, we conducted 10 interviews

○ With leaders from the AO, from the clerks advisory groups, and from key

Committees of the Judicial Conference (i.e., Court Administration and

Case Management, Information Technology)

○ In addition to the interviews, we provided the AO an open comment

period, during which additional stakeholders could contribute feedback

asynchronously

5 
 

 

 

 

Establish an empowered Product Team, 

with a Product Owner 

How to do this 

1. Establish an empowered Product Team

2. Redefine the Product Owner2 role and responsibilities

3. Establish a Product Champions group

4. Maintain Council / Committee involvement in policy decisions. Minimize their

involvement in CM/ECF decision making

5. Clarify ownership areas between Product Team, Champions, and Councils /

Committees

6. Establish and uphold communication channels between Product Team,

Champions, Councils / Committee, and the Judiciary

We strongly recommend that the AO start small, when implementing these 

recommendations; this means 1 empowered Product Team, with 1 Product Owner, 

supported by 1 Product Champions group as well as other members of the courts and 

AO. This will allow the team and organization to define success, form new processes, 

and ultimately mitigate the risk of doing too many things at once. The organization will 

be ready to scale up its efforts and devote more teams of people and attention once 

they have validation that the product is meeting user needs and the support structures 

formed are effective. 

1. Establish an empowered Product Team

This team (also known as a DevSecOps team) shares the responsibility of upholding 

and executing the product vision for CM/ECF. It is critical that certain roles are 

internally sourced, but the rest can be filled by AO employees, employees from various 

courts, or contractors. 

2 A Product Owner is a role defined within agile teams. While the AO currently has a role called Product

Owner, the context of the proposed role is significantly different – requiring specific strengths and 

responsibilities to be successful while pursuing this new and ambiguous path for CM/ECF. The AO may 

consider calling this new role something different than Product Owner, such as Product Manager, to 

differentiate between the two. An example, from the U.S. Tax Court. 

6 
 

 

 

 

 

 

 

 

 

 

 

 

 

This should be a cross-functional team capable of investigating, designing, and 

delivering code. An initial, complete Product Team might include: 

● 1 FTE Product Owner (sourced from the Judiciary)

● 1 FTE Technical Lead3 (sourced from the Judiciary)

● 1 FTE User Researcher (or UX Designer)

● 1 FTE Front-End Designer (product/visual designer)

● 2-3 FTE Developers (front and back end)

● 1/2 FTE Testing Lead (sourced from the AO)

● 1/2 FTE Security Lead (sourced from the AO)

As the product matures and the team takes on more work, the team will likely need 

support from roles such as a Business Analyst, a second User Researcher, and / or a 

Content Designer. 

Considerations 

What good 

looks like 

What to watch 

out for 

● Full-time commitment for the majority of the team

● 1 team, 5-7 people, no more at this phase4

● Clear team charter that explains the roles, contributions, and

commitment from all team members

● The Product Owner acts as a point of contact (POC) for the team to

facilitate collaboration with Champions, unblock the Product Team, and

take on feedback from additional stakeholders

● Delayed approvals and access to technology or data that prevents the

team from making progress

● User stories5 that are too rigid to allow flexibility in how the team solves

them

● The product roadmap is not validated by data from end users

● The team is prevented from choosing what it works on

3 A Technical Lead coordinates with the organization as technical problems or questions arise, and

continuously assesses technical risk, complexity, and scope for ideas that the team identifies. This role 

has unique responsibilities, but could be fulfilled by one of the Developers on the team. An example, 

from the U.S. Tax Court. 

4 A smaller team is better for collaboration. The size of the team matters, especially when the product is 

early in its design and development. A Two Pizza team, coined by Amazon, is a rule of thumb where 

teams are “no larger than can be fed by two pizzas.” 

5 A user story is a short, user-centric description of a feature. Here is some guidance for writing and 

mapping user stories to user needs. 

7 
 

 

 

 

 

 

 

● The team is prevented from designing its own agile practices and

ceremonies

Tips for 

success 

● Discuss individual objectives, goals, roles, and interests within the team

upfront, in order to align them with organizational success

● Establish collaborative practices and spaces within the team (How we

collaborate at 18F)

● Schedule regular meetings for the team to reflect on what is working

and what should change to create space for good collaboration 

practices 

● Have patience for slow progress at first. This time will serve as an

investment for toward setting up effective team structures and tooling

● Encourage everyone on the team to regularly attend user research

sessions on a rotating basis (avoid having more than 2-3 people attend

individual sessions)

● Be prepared for the team focus and makeup to change (for example,

there’s often a ton of security work needed at the beginning, to get

things set up and people trained that then tapers off)

● Create space for designers and developers to pair together on solving

problems

2. Redefine the Product Owner role and responsibilities

The Product Owner should be given the authority and responsibility to set and uphold 

the vision of CM/ECF at both strategic and tactical levels, while consistently ensuring 

that every product decision meets user and organizational needs. The Product Owner 

acts as the lead and representative for the Product Team. 

The Product Owner... 

● Owns product vision and outcomes

● Ensures high value is created with every release

○ Defines the product backlog

○ Works with business analysts (BA) to creates actionable user stories and

acceptance criteria

○ Prioritizes backlog

○ Accepts completed user stories

8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

● Needs to be available to their team, facilitates core ceremonies, user story

“sign-off,” and clearing blockers6 to the Product Team’s progress

● Understands the environment, the end user, and the business so as to make

sound decisions

● Provides effective updates about the project, tailored to different audiences

“Picking the right Product Owner is the difference between success and failure of a 

project. You need someone high enough up in the organization that they will be 

trusted to set the priorities of the project and deliver on it. But you also need 

someone with enough time to commit to the project.” 

— Senior executive at a peer agency 

Considerations 

What good 

looks like 

● Seeks user validation for value / impact and measures success based

off it

● Understands the context of the organization, the stakeholders involved

and maintains relationships / communication with them - regardless of

hierarchy (we firmly believe this role should be sourced within the

Judiciary and not outsourced)

● Gains and keeps the trust and confidence of stakeholders (especially

senior stakeholders)

● Good at making difficult decisions under pressure

● Strong communication (oral & written) skills

● Can execute on the daily work while remaining aware of the overall

scope of the project (e.g. can see the forest for the trees)

● Has the necessary time to devote; this will be a full-time job

● Aligned with Judiciary stakeholders and the Product Team on discrete

decision making authority areas

● Takes feedback into account, seeks deeper understanding of

recommendations without saying “no” upfront

● Incorporates user research findings into project direction

What to watch ● Product Owner continues to default to execution and coordination 

out for responsibilities; they are not entrusted with setting the priorities, making 

final decisions, owning the vision 

6 A blocker, or impediment, is anything that stops or slows down the delivery of a product, like a bug, 

dependency, or decision. 

9 
 

 

 

 

 

 

 

 

 

 

 

 

● Only measures success based on productivity / velocity7

● Makes decisions on their own without collaboration with the team

● Doesn’t maintain open communication outside of the team

● Focuses primarily on meeting deadlines over meeting users’ needs

● In order to influence the Product Owner or the path forward, people go

to the Product Owner’s supervisor or other superiors.

Tips for 

success 

● Choose a Product Owner who can build consensus and alignment

around the project

● Choose a Product Owner who understands the organization,

stakeholders, and users

● Establish direct access between the Product Owner, Product Team,

and a variety of users

● Establish regular and direct access between the Product Owner,

Product Team, and Product Champions

● Ask leadership to explicitly and vocally entrust and empower this

person (consider having the AO Director and leaders across the

Judiciary make a joint announcement)

● Use demos and updates to maintain good communication up the

leadership chain and out across the Judiciary – not to seek permission,

but share red flags or warning signs

3. Establish a Product Champions group

Product Champions are a mixed group of 4-5 executives spanning the AO and the 

courts, who are responsible for clearing blockers for the Product Team, socializing, and

promoting the Product Team’s work with stakeholders. It’s valuable that the people

and involvement in a Champions role remain consistent throughout product 

development. 

Product Champions...

● Clear external blockers for the team to allow them to focus on their work

● Relay product roadmap and progress to relevant stakeholders

● Advocate for funding and project prioritization

● Do not commit the Product Team to features

● Contribute to and support the Product Team’s definition of success

● Represent the goals and interests of Product Team or external stakeholders

when they aren’t in the room

7 Velocity is a productivity measurement commonly used in software development. Velocity = units of 

work, like open tasks or tickets, completed in a given timeframe. 

10 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Leadership’s role is coaching to outcomes, not making design decisions. Deciding if 

a button is blue is not a good use of their time. Their role is defining what problems 

we want to solve, how do we measure and what success is getting closer to solving 

it and if we’re getting closer. Leave the team alone.” 

— Senior executive at a peer agency 

Considerations 

What good

looks like 

● A small, cross-functional group of leaders across the organization

(equal parts AO and courts)

● 2-4 people, no more than 58

● A regular, recurring meeting, at least 1-2 hours every sprint (usually a

two-week cycle), to advise on the path forward and collaborate with the

Product Team on a sprint cadence

● The group demonstrates a united front to stakeholders

● Gives credibility to the vision, the Product Owner, and Product Team

● Delegates decision making to the Product Team by default

● Product Champion involvement tapers off as Product Team gets up and

running, and the Product Champions shift focus to other product

initiatives

What to 

watch out for 

● They undermine the authority of the Product Owner and Team by acting

as decision-makers on product-related decisions like feature

prioritization, the roadmap, and the backlog

● Product Champions communicate the vision and progress updates

inconsistently to stakeholder groups, especially when the Product

Owner or Team isn’t present

● They conflate their role with being product or technical experts on the

system, mistakenly focusing on granular details of the product or the

technology rather than the big picture

● Failure to uphold decisions consistently

Tips for ● Avoid existing power structures or titles (committees, council, working

success groups, expert panels) to allow the role and success for this group to be

redefined

● The Product Owner should keep the Product Champions well informed –

Product Champions should never hear something for the first time in a

meeting with other stakeholders

8 “The smaller the better. With more than five people you lose the singular vision on these things. You 

can’t make decisions efficiently.” – Senior executive at a peer agency 

11 
 

 

 

 

 

 

 

 

 

 

 

 

● Choose people who:

○ Are well-networked or can clear blockers to limit the layers of

approval needed outside the Product Team

○ Can be a team player

○ Are accepted influencers and thought leaders

○ Have the authority to grant approval

○ Have humility about what they don’t know or are not an expert in

What we heard 

“Defining success is the number one thing. They don’t have to be the ones to define 

it but they have to own it and have some governance to hold the team accountable.” 

— Senior executive at a peer agency 

“Create a venue where every few weeks a touchpoint with all key decision makers 

who can give top cover to 99% of the decisions that need to be made are there. 

They can make decisions on the eight things that the team couldn’t and a blocker 

was cleared.” 

— Senior executive at a peer agency 

4. Maintain Council / Committee involvement in policy

decisions. Minimize involvement in CM/ECF decision

making.

Advisory Councils and Judicial Conference committees have the authority to decide on 

Judiciary-wide policies. They will advise the Product Team and decide on these 

matters as they relate to the product being designed and executed. There are many 

different types of councils and committees, so it’ll be critical to engage with just 3 or 4 

so that the Product Owner and Product Team can meet with them regularly to 

showcase demos9 and get feedback, without getting bogged down with meetings and 

updates. 

9

In the agile spirit of “demos, not memos.” Instead of measuring progress by looking at purpose-made 

artifacts, look at the actual work that is being done. 

12 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Considerations 

What good 

looks like 

● Advising / deciding on policy or legal issues – it’s most critical that they

help navigate the tension between standardization of processes or

methods vs. critical, unique needs and find a common ground

● Provide feedback on vision, strategy and communications

● Support speeding up the pace of change (i.e., transition from shipping

code to production 2-3x a year to once every sprint, at a minimum)

● Hold the Product Team accountable for regularly delivering code / value

to users (via demos of features under development)

What to 

watch out for 

● Bias that represent interests other than end users of the system

● Group meetings turning into ideation sessions that perpetuate a culture

of design by committee

● Courts and unit executives will want to direct product decisions

● Courts, judges, or other influential stakeholders leverage their stature or

relationships to force the Product Team in a direction that favors their

needs or perspective

Tips for ● Join an existing meeting or set up recurring time to meet with each

success group, possibly 30 min - 1 hour every month, with:

○ Clear agendas

○ User research highlights and demos

○ Topics for discussion, asking for input and / or points of contact for

follow-up research

● Avoid voting sessions and decisions involving the product

● Take feedback live in the meeting or asynchronously through email / MS

Teams

What we heard 

Working groups10 take a lot of upfront work to get stakeholders together. They’re 

not representative [of end users]. We want to connect directly to users in user 

communities, panels of attorneys, court staff. Instead of having 1-3 folks, have a 

rotating list of 3-4 dozen dozen. We ask them to take a look at what we have. 

— AO stakeholder 

10 “Working groups” is used in the colloquial sense here and does not refer to a specific type of entity in 

the AO. 

13 
 

 

 

 

 

 

 

 

 

 

5. Clarify and protect ownership areas between Product

Team, Champions, and Councils / Committees

Early on, the path forward will be vague and frequent collaboration on decisions will be 

valuable. But as the Product Team begins to normalize its processes and make 

progress in its work, it’ll be valuable to establish clear lines of involvement and 

authority. 

Considerations 

What good ● Product Team owns product decisions

looks like ● Product Champions own org / resource decisions

● Councils / committees, and AO SMEs own policy decisions

What to watch 

out for 

● Politics driving feature prioritization

● Decisions being made behind closed doors, without transparency

● Confusion remaining around who is the decision maker in important

scenarios, blocking the Product Team 

● Collaboration desired on all sides, but clarity is lacking on who the final

decision maker is 

Tips for ● Outline real, critical decision making activities

success ● Define the people or groups that may be involved (e.g., which councils /

committees will advise the Champions on particular topics) 

● Identify their default level of involvement and authority

● Share this broadly within the organization and among the courts

● Revisit every 3-6 months to revalidate or update

14 
Ownership and involvement can be codified in a RACI chart 

A responsibility assignment matrix, known as a RACI chart, is a diagram used to clarify 

roles and responsibilities in project management. The role of this framework and chart 

is to clearly document levels of involvement, it’s the Judiciary’s responsibility to agree 

to these definitions and uphold them. The chart defines whether the people involved in 

a project activity will be Responsible, Accountable, Consulted, or Informed. 

Key: 

● Responsible does the work to complete a task or implement the guidance.

○ Every activity needs at least one Responsible group.

● Accountable has decision-making or veto authority.

○ Ideally only one Accountable group is assigned to each activity.

● Consulted provides advice or guidance based on their subject matter expertise.

● Informed needs to be actively kept in the loop on project progress.

The RACI chart below is meant to provoke conversation about how decision-making 

responsibilities could be clarified in order to evolve CM/ECF around the needs of its 

end users. The groups align with our recommendations above with additions of 

councils / committees representing the groups that advise on legal policy and AO Lead 

or SME representing the groups or leaders of various AO-specific divisions (Security, 

Infrastructure, Operations, etc.). Some activities offer starter recommendations. Others 

are left blank for you to continue the discussion. 

Activity Product Team Product Champions Councils / AO Lead or 

Owner Committees SME 

Set CM/ECF’s vision, 

objectives, and goals 

Accountable Consulted Informed 

Set roadmap and 

priorities 

Accountable, 

Responsible 

Responsible Consulted Consulted Informed 

Manage backlog, plan

sprints 

Accountable, 

Responsible 

Informed Informed Informed 

Understand and 

establish organization 

risk 

Informed Consulted Consulted Informed Accountable, 

Responsible 

Prioritize security fixes Accountable, 

Responsible 

Responsible Informed Informed Consulted 

 

 

 

 

 

15 

Responsible Responsible 

Responsible
Activity Product Team Product Champions Councils / AO Lead or 

Owner Committees SME 

Accountable Informed Consulted 

Consulted Responsible Accountable Informed Consulted 

Accountable, 

Responsible 

Informed Informed Informed 

Accountable, 

Responsible 

Consulted Consulted Informed 

Accountable, 

Responsible 

Consulted Informed Informed Consulted 

Consulted Responsible Responsible Accountable Consulted 

 

 

 

 

 

 

 

 

 

... ... ... ... ... 

Pursue tech-neutral ATO 

(Authority to Operate) 

process to mitigate risks 

Decide to move to a 

third party cloud 

What and when to ship

to production 

Define user groups and 

decide which to include 

Determine which tools to 

build with 

Policy decisions 

Make "enterprise-wide"

tech decisions 

Assign staff and funding ... ... ... ... ... 

... 11

11 These sections are left blank for the Judiciary to discuss and develop. The areas of the chart that are filled out are 

preliminary recommendations. It’s the Judiciary’s responsibility to agree to these definitions and uphold them. 

16 

Responsible Responsible 

Responsible 

Responsible
 

 

 

 

 

 

 

 

6. Establish and uphold communication channels

between Product Team, Champions, Councils /

Committees, and the Judiciary

As the team is beginning to build a cadence for delivery, communication channels and 

meetings should be consistently maintained to socialize progress updates, ask 

questions, and get feedback. 

Considerations 

What good ● Demos, not memos

looks like ● Working in the open12

● Create a feedback loop of frequent communication and collaboration

among all stakeholders 

● Provide regular updates

12 “Working in the open” is a common phrase within agile teams but it can have multiple meanings. 

Within this context we mean maintaining a roadmap that is visible to stakeholders, providing regular 

progress updates, maintaining documentation, and providing feedback channels (e.g., demos) for

consistent communication and collaboration. In broader terms, “working in the open” can refer to open 

source policies, which we refer to and define later in the report. 

17 
 

 

 

 

 

 

 

 

 

 

 

What to watch ● Progress updates get derailed with ideas and discussion. If this occurs,

out for set up a separate time to discuss with the people involved. 

● Feedback is given but not acknowledged or followed up on (if needed)

● Stakeholders make demands on the Product Team that change their

priorities 

Tips for ● Send progress updates to all informed stakeholders every sprint,

success consider using a weekly ship email format (source)

● Attend monthly meetings with demos, research clips or verbatims, open

questions to discuss, and anything they can help unblock 

● Keep it brief and make space for reactions

What we heard 

“Champions need to protect the Product Team from outside influence by 

proactively communicating to stakeholders about the decisions and direction of 

the Product Team.” 

— Senior executive at a peer agency 

"If we don’t include these stakeholders, they can derail everything, beyond our 

control. As a process we want to be sure to keep everyone informed. 

Transparency and communication are key.” 

— AO stakeholder 

18 
 

 

 

 

 

 

Conduct research with real users to 

direct all product decisions 

How to do this 

1. Invest in user research roles and activities

2. Set objectives and goals around user value

3. Establish more user research data sources

Who are “real users?” 

People who participate in user research activities should be regular end users from 

identified user groups, and not SMEs. The Experiment and Iterate report referred to 

these people as the “hands on keyboard” users such as court staff and public filers. 

End users don’t need special training to participate in user research activities because 

the goal is to learn from their direct experiences using the product. However, user 

research participants might become too familiar with the activities if they participate 

too often. Users should only participate more than once if the research activities or 

topic areas are different from previous research sessions. The User Researcher can 

follow up to ask participants clarifying questions about a topic or prototype. Real 

users, members of the court and public, will have the final say on what succeeds vs. 

fails or what to prioritize vs. deprioritize, via research and testing. Their voice is 

fundamental to the success of this entire initiative and should be sought and 

appreciated as such. 

⚠ Warning Sign

In the AO’s early plans for this initiative, these stakeholder groups are being positioned as 

“User Communities” in a “User Centered Product Development Process,” but most are not 

“hands on keyboard” users: 

● JCUS Committees (CACM, IT, Budget Committees)

● Public Stakeholders (Congress, Attorneys, DOJ, Public Users)

● Internal Stakeholders (Courts, Advisory Councils, CSO, CMSO, JDAO, ITSO, COO, TSO)

● Administrative Office Leadership (Director, Deputy Director, Associate Directors of DPS,

DTS, DAS) 

When stakeholder groups stand in for real users, the people with the most power get their 

goals addressed, rather than the users that have direct experience using the product. 

19 
 

 

 

 

 

 

 

 

 

1. Invest in user research roles and activities

User Researchers (also known as UX Designers) advocate for end user needs to the 

Product Owner, Team, and Champions. They regularly guide the team through 

industry-standard research activities like contextual observation, interviewing and 

usability testing. These activities allow the Product Team to understand how CM/ECF 

is or isn’t meeting end user needs and use those insights to refine the user experience 

of the product iteratively over time. The Product Team identifies and recruits a pool of 

users for research to make sure that research is not blocked by waiting for access to 

users. 

User Researchers... 

● Make sure the product works for all users

● Prioritize the input from end users, like court staff and public filers, over secondary

users who do not actively use the product

● Make sure features meet user goals. These goals are defined through research

activities like observing users doing tasks in their natural setting, interviews with

users that uncover their attitudes, and behaviors related to accomplishing goals

with the product

● Validate the need to invest in developing a feature by testing a prototype first

Considerations 

What good

looks like 

What to watch 

out for 

● Product Team participates in user research activities led by a User

Researcher or UX Designer on a regular basis (such as attending a

usability session)

● Real end users participate in user research

● Product Champions and councils / committees hear highlights from

user research

● Business goals from stakeholders knowledgeable about CM/ECF and

AO or Expert Panels replace research with end users as the way to

understand user needs biases the research process

● The Product Team uses their experience as a proxy for user research,

like trying to reproduce bugs or issues in their own environment instead

of having users show them what they are experiencing

● Research participants are chosen that are already involved in the

development process, on a council / committee, or as an AO SME

20 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tips for success ● Base all user stories and work on user research 

● Consider conducting user research on a monthly basis with one sprint

devoted to planning / scheduling and one sprint for conducting and

synthesizing, both led by a UX researcher

● Create a pipeline of users that includes different types of users from

each court in each district and public users across different roles and

geographies

● Develop a repository of user research insights that can be reviewed in

case they relate to new areas of research

2. Set objectives and goals around user value

Prioritize features and stories based on what creates the most value for real, “hands on 

keyboard” users. Success is measured in the same way. 

Considerations 

What good

looks like 

● Key Performance Indicators (KPIs) used are focused on user

satisfaction with the product 

● Additional KPIs to build healthy habits could be meeting code

standards that the team sets for itself, shipping code every sprint, etc. 

● Reducing administrative burden (time / effort / confusion)

● Organizational leaders are asking for user value indicators in

prioritization meetings and demos 

What to watch 

out for 

● Product Team and leadership continue to evaluate success based on

velocity and other productivity measures 

● Organizational leaders see a big release as success in itself

● KPIs focused on efficiently delivering code and not user satisfaction

Tips for success ● Prioritization factors based on value to the user may include: Number

of users impacted, frequency of use, urgency of use case, effort and /

or number of steps 

● User satisfaction indicators may include: overall ease and effort,

reduction of actions or content, retention of information 

● User value factors and indicators are often subjective and biased, use

them as directional indicators (and not statistically significant). 

21 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

What we heard 

“All management should be product champions. Sometimes they’re more concerned 

with driving the focus rather than supporting teams. They ask teams to pull metrics, 

metrics, metrics. I’d rather the team get work done.” 

— Subject Matter Expert 

“Demand drives action. The desire and demand for change is there. Writing good 

software is like making good food. ‘Sizzle sells the steak.’ At the end of the day, does 

the application meet their needs?" 

— AO Stakeholder 

3. Establish more user research data sources

Qualitative data from user research and usability testing findings will be valuable for 

gauging success. These sources help validate and tell stories across the organization 

to build trust. They also provide baseline metrics for future research. As the product 

begins to form, consider adding simple analytics to measure success at scale and 

staffing the Product Team with people devoted to analysis as well as design and 

implementation. 

Considerations 

What good

looks like 

● Using a variety of data points, not just specific ones

○ Qualitative: User interview quotes

○ Qualitative: Usability test videos / sound bites

○ Qualitative: Confirmation of task completion

○ Quantitative: Path to success (clicks, pages, time, etc)

● Datapoints help identify or prioritize additional research

What to watch ● Not collecting baseline data

out for ● Data is evaluated in isolation and not put into context

Tips for ● Seek a mix of qualitative and quantitative data points

success ● Correlate data points to product objectives and goals

What we heard 

22 
 

 

 

 

“We want to think about moving from points measured to value delivered to the 

outside world. Penalties for things not delivered to the outside world comes down to 

them having a metric they’re starting to pay attention to that’s not strictly internal. It 

can be a number of support requests, repetitive support requests. Are users 

successful? Are they going away happy or unhappy?” 

– Subject Matter Expert

23 
Put existing data ownership and risk 

standards into practice 

How to do this 

1. Manage risk by fully complying with the existing risk management framework

2. Develop a path forward for third party cloud

3. Support continued investment of DevSecOps tools and pipelines

4. Clarify data stewardship roles

Moving to a centralized system as recommended by the Path Analysis removes many 

of the intractable development, security, and operations (DevSecOps) problems 

associated with CM/ECF. 

A centralized system will allow the AO to: 

● Properly implement the existing Judiciary Information Security Framework

(JISF). The AO already has much of the infrastructure it needs in place to do this,

but it will require cooperation across the organization

● Develop processes to allow much faster responses to security issues such as

the recent log4j vulnerability

● Have a quicker release cycle and be more responsive to user needs

● More easily take advantage of third party cloud offerings

1. Manage risk by fully complying with the AO’s risk

management framework

The AO already has a risk management framework based on national standards: the 

Judiciary Information Security Framework (JISF). Compliance with the JISF has 

historically been inconsistent and incomplete because of the distributed nature of 

CM/ECF. Notably, the ‘authorize’ step of JISF has not been adopted, which creates a 

gap in organizational accountability and provides avenues for security and privacy 

lapses. Centralizing CM/ECF will allow these challenges, and many of the likely 

challenges associated with adopting DevSecOps for CM/ECF, to become more 

manageable. 

The Product Team should work closely with the IT Security Office (ITSO) to build in 

compliance with the JISF (including the authorize step). We expect the ITSO will have a 

24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

role in establishing security controls, in informing the AO on the appropriate selection 

of security-related roles, in training the Product Team, and in helping the Product Team 

implement automated security scans. The Product Team will be responsible for 

implementing and continually assessing conformance to security standards. 

The process defined in the JISF is cyclical, which works in tandem with the agile 

development cycle to ensure that, as new features and data are incorporated into 

CM/ECF, the system continues to meet risk management standards set by the AO. 

This process will lead to a formal Authority to Operate (ATO) gate check for CM/ECF 

and associated modules / tools before production goes live. 

Considerations 

What good ● Clear, technology-neutral risk management policy

looks like ● Product Team engages existing security and product expertise within

the AO

● Policy-based security guardrails, not technology mandates

● Product Team understands and can implement security at the

information system level 

● Common security controls are a motivation for developing DevSecOps

pipelines 

● Product Team leverages existing work of data governance groups when

categorizing data risk levels 

● Alignment on roles such as Authorizing Official and System Owner

What to watch ● A culture of non-compliance or partial compliance with security controls 

out for based on non-security pressures such as feature requests or product 

timelines 

● Mandating specific technology as a shortcut to security compliance.

The Product Team should be able to choose the tools to best meet user

needs as long as they can comply with security controls

● Product Team delegates responsibility to others for the security of their

work 

Tips for ● Provide security training for Product Team to encourage a culture of

success safety

● Begin ATO process early so Product Team can establish a security

mindset from the beginning 

● Engage ITSO early to establish common controls

● Use existing AO security expertise to develop and communicate a

holistic risk management policy that covers data governance, 

DevSecOps, and working in the cloud 

25 
 

 

 

 

 

 

 

 

 

 

 

● Frame the ATO process as a service to developers: it frees them from

having to make security decisions without the proper context and 

provides a clear framework within which to work 

● Frame the ATO process as a service to courts: it provides transparency

into organizational security and specific documentation about how

Product Team is meeting the security requirements 

What we heard 

“Security lapses go back years—admin credentials are being hardcoded into code, 

credentials are stored in projects and in Jira, Gitlab, and Access.... We need to build 

security in to avoid this.” 

— AO Stakeholder 

“Not all courts have Information Security Officers that understand the information 

security space well enough. When we come back to them and say, ‘Do you realize 

you have this risk?’ They don’t know.” 

— AO Stakeholder 

2. Develop a path forward for third party cloud

We discovered almost universal acceptance for moving to the cloud across the 

organization during our research. Third-party cloud services like Microsoft Azure or 

Amazon Web Services (AWS)13 offer compelling advantages over running data-centers 

and managing servers, including: 

● Cost transparency

● Massive scalability and redundancy

● Freeing AO resources to focus on the Judiciary’s primary mission

● Quicker development cycles and easier deployment of services (servers can be

deployed in minutes)

● Access to modern development tools like NoSQL databases, serverless

architecture, and monitoring

● Leveraging the service provider’s scale and expertise to discover and respond

to security vulnerabilities quickly

13 Azure and AWS are provided here as examples of cloud service providers, and are not being 

specifically endorsed. There are a number of options other than those listed here that the AO may 

consider. 

26 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Additionally, Federal Risk and Authorization Management Program (FedRAMP) certified 

cloud providers implement common security controls which eases the burden of 

developers when implementing a risk management framework. 

Considerations 

What good

looks like 

● Product Team is comfortable with and knowledgeable about working in

the cloud 

● Broad alignment around cloud solutions among stakeholders

● Product Team is cross-functional and works across silos

● Common inherited controls between JISF and FedRAMP providers are

aligned and documented 

● A small slice of work initially deployed to the cloud

What to watch 

out for 

● Moving data to the cloud that has not been properly categorized for the

correct level of risk 

● Cloud providers that cannot offer data security and privacy guarantees

required to work with court data 

● Spending resources and time to implement on-premise solutions when

mature cloud options are available 

● Waiting until there is a comprehensive ‘Cloud Strategy’ before beginning

Tips for 

success 

● Existing data governance work is incorporated into the system to ensure

court data is managed for risk as the Judiciary moves to the cloud

● Understand costs in terms of capital costs vs. operational costs. Cloud

pricing is relatively simple; the cost of running data centers is not

However, they are difficult to compare because servers are often treated

as capital costs, while cloud provider costs are treated as operational

● Begin with a small slice of work in the cloud. This should not be

expensive and can begin with an iterative approach to reduce risk and

get started quickly.

● Give Product Team freedom to stand up development environments in

the cloud for testing and learning 

● Closely evaluate offerings from services providers to ensure they can

meet the specialized data privacy and security guarantees needed by the 

courts 

27 
 

 

 

 

 

 

 

What we heard 

“I want us to be able to take advantage of tools available in the cloud, but I need to 

make sure it doesn’t bypass security.” 

—AO Stakeholder 

“Amazon does it [patches system vulnerabilities] in ten hours; we’re lucky if we can 

get a patch out in 30 days.” 

— Subject Matter Expert 

“Yes, the pipeline the DevSecOps Ad Hoc Working Group is building is compatible 

with the cloud. We made sure everything we picked was cloud ready, so we’re not 

re-inventing the wheel.” 

— Subject Matter Expert 

3. Support continued investment in DevSecOps

pipelines

18F’s Path Analysis and Experiment & Iterate reports highlighted the importance of 

adopting DevSecOps practices. Through the Enterprise DevSecOps Ad Hoc Working 

Group (DevSecOps WG), the AO has started the work of developing pipelines and tools 

to help developers adopt these practices. This work is valuable and should continue. 

To make the work of the DevSecOps WG most effective, the AO should allow the 

Product Team to choose their own tools and develop their own solutions, as needed. A 

Product Team will often have needs that can’t be anticipated by pipeline developers. 

When the team is empowered, they will choose common tools and pipelines that make 

their work easier by providing things like inherited security controls, automated security 

scans, automated deployments, and integration with source control, while still offering 

the flexibility in the choice of development tools. 

Like any other product, the development of these tools should use user-centered 

design methods and closely involve developers. To do otherwise risks building 

complex pipelines that will be difficult or frustrating for developers to use. 

28 
 

 

 

 

 

 

 

 

 

 

 

 

Considerations 

What good 

looks like 

● Measurable KPIs such as lead time, deployment frequency, and mean

time to recovery

● Understanding that tools and pipelines are not a substitute for a

DevSecOps culture

● Pipelines that work seamlessly with third party cloud providers

● Developers choose to use the AO-provided pipeline but have the

freedom to use other solutions 

● Automated tests and security scans are built in

● Product Team working on actual product are included in the process of

developing pipelines

● Ability to quickly deploy security patches and roll back changes

What to watch 

out for 

● Building pipelines that don’t meet specific developer needs

● Creating pipelines as technology / security mandates

● Overvaluing consistency of tools rather than responding to developer

needs 

Tips for success ● Communicate value of pipeline, especially inherited security controls, to 

developers and encourage adoption 

● Adopt user-centered design when building pipelines

● Encourage Product Team to provide input to teams working on

DevSecOps pipelines

● Provide clear documentation of pipelines and DevSecOps tools to make

them easy for developers to use 

29 
 

 

 

4. Clarify data stewardship roles

The National Institute of Standards and Technology (NIST) Special Publication (SP) 

800-39 defines the role of “data steward” and “information owner” synonymously as:

“An agency official with statutory or operational authority for specified 

information and responsibility for establishing the controls for its generation, 

collection, processing, dissemination, and disposal.” 

This role is distinct and independent from the system owner, who is responsible for the 

development, operation, and maintenance of the system. The roles, however, are 

compatible. NIST SP 800-39: 

“A single information system may contain information from multiple information 

owners / stewards. Information owners / stewards provide input to information 

system owners regarding the security requirements and security controls for 

the systems where the information is processed, stored, or transmitted.” 

We anticipate that developing products flexible enough to meet the diverse practical 

and statutory data governance needs of the courts will be a major challenge for the 

CM/ECF Product Team. There is already alignment across the Judiciary that the Clerks 

of the Courts are the legal custodians of court data. In a centralized, AO-managed 

CM/ECF, the AO and courts will need to decide what data is owned by individual 

courts versus the AO, even if it requires the AO to request statutory changes. 

Historically, agreeing on data governance and stewardship standards has been 

challenging. It will be important moving forward both for the AO to support the data 

governance needs of specific courts and for the courts to accept the AO as the system 

owner. The AO and courts have already started to categorize risk for different data 

types through the efforts of groups like the Judiciary Data Working Group (JDWG). This 

aligns with the “categorize” step of the JISF and will be valuable as the Product Team 

assesses the correct risk level of their work. 

Considerations 

What good ● Continued alignment with prior guidance that “clerks maintain control of

looks like the docket when a centralized server is used for data storage as long as

they maintain the legal right to access the data.” 

● Existing data governance work from JDWG is incorporated into the

30 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

system to ensure court data is managed for risk 

● Courts are willing to participate in developing a new system

● Current law and Judiciary policy is clear and documented

● Sufficient safeguards are in place and documented that satisfy the legal

duty of courts 

● Controls have been identified that are appropriate for the level of data

risk and are understood by the Product Team 

What to watch 

out for 

● Misunderstandings regarding laws and regulations used as a blocker

● Misalignment around the distinction between system data (like logs),

managed by the system owner, and court data owned by the courts 

● Product Team underestimates the challenge of implementing security

that accounts for the diversity of data governance needs 

Tips for ● Clarify and document the distinction between the system owner and the

success data stewards 

● Set expectations from the beginning that addressing the diversity of

court data will be a major challenge that the AO is committed to 

meeting 

● Begin development work with small set of low-risk / publicly-available

data 

What we heard 

“There’s a very vocal minority group that is very much against the AO doing 

any central administration or enterprise systems because they feel they have a 

legal mandate to own the data and be responsible for it.” 

— Subject Matter Expert 

“Just getting to an on-prem data center was a process; folks wanting the data 

center in their courthouse.” 

— AO Stakeholder 

31 
 

 

 

 

 

 

 

 

 

Start building the solution, incrementally 

If the AO believes CM/ECF is not sustainable, they need to start building something as 

soon as possible. As the AO gets started, it should consider the following: 

1. Start small to build trust

2. Create a team to begin the work

3. Set yourselves up for success in procurement

1. Start small to build trust

Starting small validates that actual user needs are being met by what is being 

developed. The Product Team pursues a small “slice” or increment of functionality, by 

working with one set of users on their most critical needs and pain points related to a 

core business process and then expanding out to meet the needs of other users. 

“When we build, we aim to release early and often to end-users using agile 

development methods. Ultimately, the government’s investment should be 

measured in working software, not phases, documents or milestones. Only 

working systems are of value to real users.” 

—18F’s Path Analysis report 

⚠ Warning Sign

The AO’s early plans for a product development journey perpetuates the current top-down 

planning process, rather than empowering a team to start building something small. A 

problematic journey might include the following steps: 

● User Communities identify priorities

● The AO Director consults with councils

● AO leadership approves resources, assigns responsibilities, establishes timelines and

success criteria 

● Product manager works with product owners, users, Product Team(s), and delivers

product 

32 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Considerations 

What good

looks like 

● Build a piece of functionality that crosses user interface, business logic,

database, and architecture

● Choosing a slice that has high user value and low technical complexity

● Product Team regularly demos, every sprint cycle, progress to

Champions and councils / committees to build trust, at least once every

three weeks (or more often depending on the length of a sprint)

What to watch 

out for 

● The small slice is prioritized by leadership and not based on user value

● The small slice use case is too complex – technically complex, not

focused on a specific tactical need or “job to be done”

● Skepticism that there is sufficient similarity between courts to allow

standardization. There are several different common ways of doing a

core business practice when there should only be one

● Groups jockeying to be the first court to participate can add politics and

bias against them as participants

Tips for ● Understand the “why” of their need so the Product Team can creatively

success develop a solution to address the outcome desired

● Start small with a group that is willing to try something new to lower

barriers to getting started

● Champions should communicate expectations with stakeholders that

starting small will mean that the needs of all users will not necessarily be

met, initially, and that the process might require patience as the needs of

more and more users are addressed through an agile iterative

development process

● Take advantage of cloud development environments, which offer low

cost and quick iteration

What we heard 

“This team will need to prove to the community that has very low trust that this 

process will work. That we actually can make this happen and make 

adjustments in the process along the way. Making course corrections is 

important.” 

— AO Stakeholder 

33 
 

 

 

 

 

 

 

 

 

 

 

 

 

2. Create a team to begin the work

18F recommends standing up a small, cross-functional team to

incrementally build a new, custom solution that leverages open

source software. 

18F believes building a new system from the ground up that leverages open source14

software is the best path forward for creating a new case-management system that will 

meet users’ needs. 

Considerations 

Why do this ● This approach invests in a team to design and build a solution that meets

the AO and courts’ unique user and organizational needs

● Unexpected changes are easier to adjust to when working in an

incremental way and with full control of the release schedule

● It is easier to verify supply chain security of open source software and

software the AO wrote because the source code is available to inspect

What good

looks like 

● Follow the recommendations in this report needed to support building

new initiatives

● Hold information sessions about this approach with various stakeholders

and court staff for feedback and suggestions

● Form or hire a new Product Team, as described above

● Identify 2-3 high value use cases to begin user research

● Solicit 2-3 pilot courts

● Decide on an application and database deployment strategy

● Start to to prototype a use case applying a user-centric, agile

methodology

What to look ● Estimating total cost before getting started, without a defined scope, it’s 

out for difficult to estimate time and cost for this work 

● There is risk that third-party open sources libraries will not be maintained

for the life of the system

● Courts don’t trust the AO to successfully build and maintain a new

solution in-house

14 18F defaults to working in the open; here is the 18F open source policy. Other agencies have 

defaulted to this as well, “what agencies have to say about working in the open.” 

34 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

3. Set yourselves up for success in procurement

If the AO seeks to procure vendor support to help them execute the vision of CM/ECF, 

we believe there are important activities to conduct in order to meet the needs of the 

users and organization. 

Only 13% of large government software projects are successful. Be 

prepared to help your organization reduce risk by implementing modern 

development practices. 

— 18F De-Risking Guide

If procuring a vendor to build a system tailored to your needs 

Considerations 

What good

looks like 

● Bring in end users early and prioritize people who interact with it daily to

determine user needs 

● Create a draft RFQ and release it for comment from vendors utilizing the

RFI process 

● Consider vendors with established contractual vehicles on SEWP or

MAS / GSA Schedule 70

● Use modular contracting and the agile contracting format

● Embody agile principles in the creation of a software development

solicitation

● Prepare your organization for product ownership and healthy vendor

management 

● Form a successful partnership with the vendor team with open

communication 

● Engage in prototyping to test concepts

● Conduct sprints to create code, adjust as necessary, and ensure vendor

performance 

● Have continuous user engagement

What to watch 

out for 

● No collaboration between vendors and the AO

● Micromanaging or asking for status reports in lieu of healthy, consistent

communication 

● Penalizing rather than assisting and rectifying the situation

● Conducting a traditional waterfall approach FFP (Firm Fixed Price)

contract for a base plus 4 years. 

● No user testing during development and the final end product by the

vendor 

● Vendor lock in

35 
 

 

 

 

 

 

 

 

 

Tips for ● Utilize agile processes

success ● Use a labor-hour or time and material (T&M) contract

● Engage agency staff to participate in DevSecOps from the beginning

● Use open source software

● Keep the period of performance at or under 3 years

● Make sure your contract has the ability to terminate the vendor for non

performance 

○ If your vendor is not meeting sprint deliverables consider termination

if the issue cannot be rectified. It is better to reassess the project 

vendor than spend millions on a project which does not meet your

needs. This is the benefit of the T&M contract.15

15 With a modular, T&M contract format the agency can issue a notice of concern, cure notice, and 

terminate for non performance if they are not meeting the requirements of the contract regardless of the

contract type. More importantly, it also allows for much easier ways of ending the contract if the 

direction of the work changes or the contractor team is not producing quality software—if their work is

inadequate, or their skills prove inappropriate, then no further work need be assigned to that contractor 

(effectively terminating the contract), and the contractor can be replaced. Sample of determinations and 

findings (D&F) for a T&M contract. 

36 
Conclusion 

During the Path Analysis and Experiment & Iterate phases, 18F learned a great deal 

about the challenges the AO faces as well as the tremendous strides the organization 

has made in meeting those challenges. One of the biggest challenges is yet to come: 

implementing a new case management system for the Judiciary that will serve the 

needs of every user, while making the substantial organizational changes that will be 

required in order to make that work possible. This report turns the findings and 

recommendations from 18F’s prior reports into concrete next steps the AO can take to 

help build that new system and adopt supporting practices. The tactical 

recommendations in this report will not be successful without the strategic intention 

behind them and the ultimate goal of delivering a product that meets user needs. 

While this report lists many steps the AO can take towards a new CM/ECF, the 

most important step to take is the first one, and the AO should take it 

immediately. 

37 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix A: Summary of 

recommendations 

Activity Tactical Recommendations 

Establish an empowered 

Product Team, with a 

Product Owner 

● Establish an empowered Product Team

● Redefine the Product Owner role and responsibilities

● Establish a Product Champions group

● Maintain Council / Committee involvement in policy

decisions. Minimize their involvement in CM/ECF

decision making

● Clarify ownership areas between Product Team,

Champions, and Councils / Committees

● Establish and uphold communication channels between

Product Team, Champions, Councils / Committee, and

the Judiciary

Conduct research with ● Invest in user research roles and activities

real users to direct all ● Set objectives and goals around user value

product decisions ● Establish more user research data sources

Put existing data ● Manage risk by fully complying with the existing risk

ownership and risk management framework

standards into practice ● Develop a path forward for third party cloud

● Support continued investment of DevSecOps tools and

pipelines

● Clarify data stewardship roles

Start building the ● Start small to build trust

solution, incrementally ● Create a team to begin the work

● Set yourselves up for success in procurement

38 
Appendix B: Unresolved Questions 

These are feedback questions from the AO and the courts about our 

recommendations. We have not investigated these questions in depth, so we could not 

answer all of them. The answers we are able to provide are provisional and tentative. 

Most of these do not need to be answered upfront and do not block starting 

work. Instead the Product Team and the AO should continue to raise these questions 

and collaborate on answers to them, incrementally. 

● How will this initiative be funded? Who will have decision making authority

over funding? For example, how does the AO manage budget projections and

funding sources for cloud services? Is the Product Owner and Team responsible

for cloud costs or will the costs be borne by the infrastructure group?

○ This should be answered by the AO, we cannot advise on funding

sources. That said, we do recommend that you fund this effort

incrementally. For example, procure cloud services in a $1-10K range so

that the Product Team can estimate or validate usage estimates for a

much larger procurement later on.

● The Judiciary as a whole needs to agree on a set of standard processes or

workflows for the system and where customization is warranted. “We can’t

design for everyone. We’ll never get it done.” – AO Stakeholder

○ The Product Team, Champions, and the Councils / Committees will need

to address these use cases as they come up.

● If a business priority conflicts with a user need, or is made without user

input, what happens? Will the Product Owner and Team be supported to make

that call, even if it means delaying the decision to get user input?

○ The Product Team and Champions will need to address these use cases

as they come up.

● When should the AO develop a strong prioritization and feedback process

to ensure users get the most valuable functionality that meets their needs?

○ We advise strongly against pursuing this too much upfront and instead to

start building something. The prioritization and feedback process should

develop organically as the Product Team starts conducting user research,

providing demos, performing user testing, etc. to deliver the product.

39 
● How does the AO develop a user research process? Who is responsible for

that?

○ The 18F UX Guide is an excellent resource for building out an approach

for research. In small teams, the User Researcher typically is responsible

for developing and leading the process. There are best practices and

common phases of this work, but research plans and approaches should

be unique to the goals you hope to achieve (whether that be developing a

journey map, identifying pain points, designing a solution, testing

concepts, etc.).

● How does project documentation (not product documentation) fit into all

this? i.e., to be transparent to the AO and courts?

○ Consistent, open communication and documentation is imperative.

We’ve made recommendations on potential channels for that, but where

it’s hosted should be based on where it’s accessible to all stakeholders

and how easy it is to maintain. Our default is to work in the open. 18F

teams have used public GitHub wiki’s, to host project documentation so

that the team, organization, and the public stay updated on progress. A

great example is this github wiki that 18F developed with the DOI’s Office

of Natural Resources Revenue for their Natural Resources Revenue Data

website.

● How are tech, security, and business priorities weighed against each

other?

○ User value should be the lead determinant of priorities, especially early

on. Then considerations around tech, security, business value, and policy

constraints add to the feasibility assessment of each priority. This should

be driven by the Product Team with contributions from the Champions,

Committees / Councils, and various Subject Matter Experts. As noted in

previous questions, we do not recommend codifying a rubric for

prioritization early on. See Using agile and DevOps to get better results

than a change control board as an additional example.

● What provides guardrails for the work of the Product Team? Such as, what

do they base their assessment of security risks on and when do questions

need to be raised to the Champions?

○ As the Product Team develops and maintains regular communication with

the Champions and Council / Committees, that should create the

40 
opportunity to identify policy-related considerations and put them in 

contact with a subject matter expert. Then it’s the Product Team’s 

responsibility to seek out more information and collaborate with relevant 

folks to find a solution that meets the needs of users. 

● If we are regularly presenting new functionality to stakeholders and the

courts, we will be getting a lot of feedback. How can we manage this

feedback to ensure we are learning what we need?

○ The Product Team should be seeking feedback from end users regularly

through testing and other means. If a Product Team is demoing their

work, we fully expect feedback to be taken from various stakeholders.

However, feedback is typically 1 instance from 1 person’s perspective.

It’s the Product Team’s responsibility to investigate further (consider the 5

why’s as a root cause analysis method) and identify common patterns. At

that point the Product Team should discuss and consider it in their

backlog.

● Could we have a legal / policy person included as one of the “users”?

Stripping away the tools, what steps need to be done at a basic, legally

required minimum?

○ If there are any legal requirements for the system, the Product Team

must be informed of that information. The capacity in which a legal /

policy person would contribute to the development of the product would

depend on whether they were participating as a SME or a user as

described in the Conduct research with real users to direct all product

decisions section. Strictly speaking, unless such a person was actually

using the system, they would be a SME regarding law / policy.

41 
 

 

 

 

 

 

 

 

 

 

 

Appendix C: How to Proceed with COTS 

This report has highlighted several points of contact between 18F’s recommendations 

and work that is underway at the AO. However, one area where 18F and the AO 

disagree substantially is on the question of whether to build new using custom 

development that leverages open source software or to pursue a COTS-centric 

approach. 

AO’s current approach is focused on procuring core COTS systems to 

lay the foundation for hosting and data, and then standing up a team

to configure or build on top of it. 

Considerations 

Why do this ● A COTS software solution is highly mature in the marketplace, and

similar organizations have successfully implemented the solution (with

“success” determined by end users, not by the vendor).

● The agency will modify its existing practices to work within the

limitations of the COTS software

● Enterprise rollout of the new solution may be quicker once initial

functionality is in place

● Time required for review and approval of technology architecture may

be significantly shorter than for a custom solution

● Software operation and maintenance is owned by a third party vendor

and they are held accountable by service level agreements (SLAs) and

responsible for fixing security vulnerabilities

What good ● Reduce investment in current platforms

looks like ● New product development process and technology architecture

● Start building new modules / features

● Deprecate and subsume old functionality

● Track progress and iterate as needed

What to watch ● Solution vendor may have development priorities that are different than 

out for the AO’s 

● The long-term viability of third parties can not be guaranteed; what

happens if they stop supporting the software, go bankrupt, are

purchased, etc.?

42 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

● As the AO begins to adopt and use it, user needs will mature and the

flexibility to address them will be constrained by the capabilities of the

system

● There is a significant risk of lock-in (a/k/a vendor lock) when choosing

COTS solutions. If the solution does not solve all user problems in the

future, it is difficult to back out without starting over

● Responding to unexpected changes is harder when major functionality

depends on a vendor’s roadmap and release schedule

● Verifying security and privacy of closed source software is difficult

● Procuring a COTS solution or component (like cloud services) upfront

for the entire organization is an expensive and risky decision to make

without validation from users over time

● Total cost of ownership may exceed initial estimates

If pursuing COTS, engage users to vet and test systems during 

procurement 

Involve a diverse set of users to formulate the evaluation factors for solicitation to 

ensure it meets organizations needs. 

Sandboxing – The only way to know if something works is if you can actually try it 

out. A sandbox environment provides an unrestricted trial of a software product. In 

your solicitation, request sandbox environments and design usability tests to run with a 

set of users. If you can’t get a sandbox environment, you can do further evaluations 

with demos, but that will require more specific requests and coordination ahead of 

time. You’ll get less accurate information and user feedback from a demo than from a 

sandbox test. Either approach should be as structured as possible to make 1:1 

comparison between offerings easier. 

Considerations 

What good

looks like 

● Bring in end users early and prioritize people who interact with it daily

● Tool meets required functionality, works with current systems

● Keep everything uniform in the sandbox experience so that results are

meaningful 

● Use the same structure when evaluating the sandbox (see Appendix D:

User Scorecard)

● Know that no COTS software is going to meet 100% of user needs;

align on a reasonable expectation for making a decision 

● Continue to involve users in the decision making process

43 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

What to watch 

out for 

Tips for 

success 

● Users aren’t involved in evaluating all COTS options being considered,

meaning they’re evaluating sandbox A and not sandbox B 

● Users evaluating products aren’t real end users

● Evaluating products against a list of requirements versus having users

use the product 

● Group evaluating all options isn’t consistent, with providing an accurate

picture of the functionality of the product 

● Features that aren’t core to your process or mission

● Make sure all sandbox presenters understand that this isn’t a formal

solicitation 

● Ensure organizational leads prioritize user perspective in the decision

making process 

● Recruit a cross section of users reflecting various roles, levels of

involvement, and organizational knowledge to ensure a holistic

evaluation 

○ Patterns can usually be seen across 5-6 users. For CM/ECF users

consider focusing on where there are common court processes 

● Keep the sandboxing team small and diverse

● Build out user stories and use them as sandboxing tasks along with

priority evaluation criteria 

● Timebox the evaluation period to about 2 weeks

● Keep potential vendors on task for sandboxing / demo and avoid sales

presentations 

● Uniformity is important: if you do move forward with demos, define a

strict, uniform structure for vendors to follow, showing how their 

product supports the exact same tasks completed from the same 

users’ points of view. For Example: 

○ Demonstrate how end users can make granular changes to

document permissions and visibility

○ Give a detailed explanation of your API capabilities and experience

integrating with other COTS products as well as custom, in-house

tools.

44 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D: Example of a COTS 

Sandbox User Scorecard 

Tasks should be edited to reflect user type, priority user needs and solicitation 

evaluation factors. Examples below. 

Tasks + Features 

(examples below) 

Score 

Score this task or 

workflow on how well it 

seems to fit your needs: 

flexible, easy to use, 

tailored to your needs 

Comments 

questions

Open field to clarify 

your score 

Weight

After the demonstration, 

numerically score the tasks

and features you saw based

on importance to you 

Start a new case, appeal,

and / or interaction with 

the system 

1 2 3 4 5 6 7 8 9 10 

N/A 

Generate a report 

Receive reminders and 

notifications 

Validate users for access 

Track and report access 

and / or issues 

Process court fees 

Support fine-grained 

access controls for sealed 

documents 

Allow data-sharing across 

court types 

Support large attachments 

Software vendor has zero 

visibility into court data 

45 
Weighting and aggregating user scorecards 

● When you’ve finished all demos or sandbox tests, aggregate the scores from

users to see patterns and identify open questions or major concerns.

● When you aggregate user needs, you can add weight to each task based on

how users have individually scored the priority of tasks, or by creating an overall

weight that is applicable to the service you chose to procure. (For example, if

you choose to procure a system that doesn’t include EDC capabilities, you can

remove tasks associated with that). You can remove all scores that are marked

“N/A”.

● After additional demonstrations, you can evaluate which solutions to pursue by

scoring them on a few additional categories relevant to your evaluation criteria,

like integration capabilities, ongoing support, training and education, and risks.

46 
Appendix E: Example of an Overall 

Sandbox Evaluation Scorecard 

This scorecard should be edited based on the AO and courts evaluation criteria. 

COTS 1 COTS 2 CUSTOM BUILD 

Integration capabilities

and support 

Ongoing support +

education 

User research, feedback, 

assessment? 

Rollout timeline 

Security requirements 

Education and training 

support 

Overall aggregate user

perception score 

User score for priority task 1 

User score for priority task 2 

User score for priority task 3 

Major benefits 

Major risks 

AO notes 

Key user notes 

47