OVERVIEWThe Service Delivery Review: Process and Technology is a flexible vendor management program designed
to augment an ongoing or completed convergence initiative. The SDR combines a customized
questionnaire with quantifiable measurements and an onsite review to produce a scorecard andrecommendations in areas the client selects (e.g., project management, billing hygiene, staffing,training, knowledge management).
With people and pricing in place, process offers clients and law firms the most levers to work
collaboratively in driving continuous improvement. The SDR creates the opportunity for an on-going
structured dialogue about increasing performance in areas of emphasis. The SDR is as much aboutcarrots as it is about sticks. The optimal outcome is a deeper relationship in which clients get better
quality at lower cost while firms increase realizations and profits.


ClientsoGain a more comprehensive understanding of how legal services are deliveredoGenerate empirically-aided opportunities to communicate with firms about prioritiesoImprove quality and reduce costs in a cooperative, systematic manner

Firms
o

Differentiate themselves from competitors and strengthen relationships with clientso

Get credit and realize a return on investments in process and technologyoAddress the presumption of inefficiency that leads to internal and external writedowns

Dive directly into the mechanics here. An annotated exemplar questionnaire is here. Sample reports
are here. An introduction to Basic Technology Benchmarking, which addresses common officetechnology like Word and Excel, can be found here.

Philosophy
Strategic sourcing is an approach to supply-chain management premised on the inherent value in longterm, mutually-beneficial relationships. The hard and soft costs incurred in switching suppliers afford
advantages to incumbency. Exemption from reasonable scrutiny, however, is not among them. Rather,strategic sourcing calls for a rigorous but collaborative approach to continuous improvement across theentire value stream.

Legal acumen is mission critical. The operating assumption is that the organization already hires superb
lawyers both internally and externally. But the provision of legal services is not solely a matter of
abstract legal insight. While the insight is primary source of value, execution matters, especially forconsistency and cost. Much of the labor (and waste and error) comes in translating abstract insights into
concrete deliverables such as contracts, closing documents, filings, etc. Expanding the focus fromindividual aptitude to the mechanisms by which legal services are delivered substantially increases thelevers available to drive continuous improvement.
The primary purpose is not to punish incumbent law firms for operational inefficiency but to build deep
supplier relationships that move continually towards operational excellence. This starts with committing
to coprosperity and learning about how the suppliers works.
The model for deep supplier relationships is the Japanese automakers in comparison to their American
Big 3 counterparts a decade ago. When the competition from the Japanese intensified, the Big 3 studied
the Japanese cost structure. They found that the supply base was a major source of Japanese costadvantage. The Big 3 tried to close the gap by leaning on their suppliers for cost reductions. The Big 3
achieved cost reductions. Just not enough. They also had to deal with inferior quality parts anddecimated, antagonized suppliers, many of whom eventually went bankrupt.
The Japanese automakers responded to the competitive pressure by setting cost reduction targets of
their own. They achieved cost reductions. At the same time, quality improved, and the Japaneseautomakers deepened their relationship with an engaged, profitable supply base. Their suppliers,
including American companies, were able to fulfill the twin mandates of cost reduction and quality
improvement because the Japanese automakers helped them do so. The Japanese automakers
dispatched teams of consultants to map supplier value streams, identify high-impact process
improvements, and implement improvement plans in a sustainable manner. The Japanese automakers
were just as hard on their suppliers, if not harder, than their American counterparts. But they
approached their suppliers in the spirit of active engagement rather than antagonism.

The American legal market has grown more antagonistic since the Great Recession. As reflected inplummeting realization rates, inside counsel already ‘know’ their outside counsel are inefficient. But
these are imprecise impressions of poor service delivery, not a concrete identification of remediableproblems. Law firms therefore have limited opportunity to demonstrate value and alleviate client
concerns. Clients do not reward process improvement with higher realizations or additional work. With
no return on investment in innovation, stagnation reigns and client discontent deepens.

Transparency benefits both sides. A client that understands the mechanics of service delivery can set
clear expectations for improvement. Rather than a vague, unilateral mandate, these expectations should
serve as a foundation for a structured dialogue about priorities, timelines, milestones, and measurable
results. The interaction should run both directions, with the law firms also identifying the ways in which
the client can help facilitate the delivery of superior legal service.
There is no finish line. Individual projects will run their course. There will always be more levers to press.
Progress, not perfection, is the objective. Progress includes the alignment of interests between clientand firm through structured dialogue and continuous, mutual improvement.

Mechanics
Step 1: the (external or internal) consultant conducts interviews with the client to customize thequestionnaire and scorecard, including category selection and weighting. In addition to the managing
attorney, interviewees should include operational and administrative personnel. An annotated,
exemplar questionnaire is appended here.
Step 2: the consultant works with the firms to complete the questionnaire, responding to inquires about
the questions themselves, as well as the form and content of the answers.
Step 3: the consultant conducts follow-on interviews with law firm personnel. These interviews include
not only the relationship partner and other timekeepers, but also their support staff and nontraditional
stakeholders such as the information services, knowledge services, project management, and pricing
professionals. The interviews will be driven by responses to the questionnaire as well as an onsite
review protocol customized to the client. An exemplar onsite review protocol is here.
Step 4: the consultant presents preliminary findings and recommendations to each law firm forcomment and feedback. Exemplar findings are here.
Step 5: the consultant finalizes the findings/recommendations for each relationship and provides themto both the client and the law firms. The consultant provides the client with a comparative report and aconsolidated response document, which contains all the firms’ responses, as well as the consultant’s
notes from the onsite review. An exemplar comparison is here.
Step 6: the client and the law firm meet to discuss the findings and recommendations. The consultant
participates at the beginning of the meeting to set stage and at the end of the meeting to document
commitments, deliverables, timelines, etc.
Step 7: repeat (annually or biannually) with an ever-evolving questionnaire that uses previous
findings/commitments as a baseline and is modified to accommodate changed circumstances and newpriorities.

Other Considerations
How many law firms? The process works with just one firm. That said, relative rankings tend to be more
informative than evaluation against an abstract standard. Multiple firms answering substantially similar
questionnaires also introduces economies of scale. If, however, the process is to be used with more than
five firms, the recommendation is to create rational subgroups for suitable comparisons andadministrative ease.

Which law firms? The law firm relationships with sufficient incentives on both sides. The relationshiphas to be valuable enough to the firm for them to invest in process improvement. And the relationshiphas to be valuable enough to the client to invest time in understanding the firm’s delivery mechanism
and communicating clear expectations about process improvement.
For firms that don’t quite meet the threshold but are still substantial partners, the recommendation isfor the client to make a conditional commitment to act in concert with clients in a similar circumstance.
That is, the client commits to be one of the X number of clients that will request responses to a moregeneralized questionnaire. It is up to the consultant to find the critical mass of other clients with asimilar level of interest.
There is also merit in a scaled-back SDR for firms with less strategically vital relationships. Generalquestionnaires with limited follow-up and no onsite meeting can be used for evaluative purposes as partof a broader vendor management initiative.
Who pays? The firms probably should. This is marketing in the purest sense (to know and understandthe customer). The results are also portable. The firms can use positive findings as selling points withother clients.
The marketing angle is particularly apt if multiple clients are asking after the same information (e.g., theconditional commitment scenario). If there is only one client asking, there is more of discussion to be
had about the client covering some, if not all, of the consulting costs.

Sample Findings

Sample Comparison

Annotated Questionnaire
Basic User Hardware/Software
Reasoning: this section is aimed at establishing a very simple baseline for whether the law
firm invests in technology. There will be no granular distinctions about which new computeror smartphone is best. The object is simply to ensure that timekeepers are not stuck on
obsolete hardware or software that makes it hard for them to do their job.
Rubric: on the one end will be the laggards with timekeepers running old programs on oldcomputers. On the other end will be firms who provide modern software on modern
hardware with a productivity-enhancing workstation configuration.

1. What is the most common hardware configuration for users across the firm? What options dousers have?
Example: they can select between a Dell Desktop OptiPlex 3020 or HPZbook 15 G2 laptop, andbetween the iPhone 5S and a Galaxy 5 for their smartphone

2. What is the most common operating system and default software packages? What options dousers have?
Example: Windows 7, Office 2010, and Acrobat X. They can upgrade to Acrobat X Pro upon request.
In addition, there is a process in place for individual upgrades. The IT department permitted 183 of
the 224 partner-approved requests last year. The granted requests included Microsoft Project,
Visio, and WordRake.

3. What is the most common monitor configuration (number and size) across the firm? What
options do users have?
Example: laptop with a 15.6” screen connected to 23” monitor. A second 23” monitor is available
upon request.

Document Management/Mobility
Reasoning: this section is aimed at flexibility and freedom from frustration. Again, there will
be no granular distinctions preferring one contemporary option over the other. The object
is simply to ensure that timekeepers have easy access to their work no matter where theyare.
Rubric: on the one end will be the laggards with lawyers running obsolete versions ofstandard software and no meaningful mobility options. On the other end will be firms whooffer their timekeepers a seamless, mobile work environment.

4. What document management system (including version) does the firm utilize?
Example: Opentext Connectivity 14

5. What remote access options (including software versions) does the firm offer?
Example: Citrix XenClient 4.6

6. What hardware, software, and data connections does the firm provide to timekeepers to ensure
they can work when mobile?
Example: every timekeeper has a VPN-enabled laptop and a mobile phone (iPhone 6 or Galaxy S6)
with mobile hotspot capability enabled and unlimited data plans.

Staffing Ratio/Delegation
Reasoning: one convenient but unconvincing argument against the need to emphasize
process and technology is that lawyers have plenty of staff to whom they delegate. Not
only does argument this ignore the quality degradation that comes from the resulting
inefficiency (communication breakdowns) but it also assumes that staffing ratios haveremained constant. Yet, as many have publicly stated (see, here, here, here, here, here,
here, and here), firms are aggressively substituting technology for staff.
Rubric: this will be relative. In part, surveyed firms will be compared against each other.
But it will also be about how the firm approaches process and technology. A case can bemade that well-trained lawyers utilizing good technology within a well-designed workflow
require less staff support. Finally, the import of the numbers will be client and area specific.
Sometimes, a client may expect little leverage—i.e., they want most of the work done at
the partner level. There will be other instances where the client expects the partners todelegate as much work as possible to lower cost or no cost (to the client) personnel.

7. What are the firm’s ratios of timekeepers (e.g., partners, associates, paralegals) to the nontimekeeping professionals to whom client work is delegated (e.g., secretaries, word processing,other support staff) excluding back office functions like accounting, marketing, IT, andprofessional development?
Example: the firm has 110 partners, 270 associates, 70 secretaries, 20 paralegals, 10 word
processors, an 8-person office services staff, and 2 librarians. The ratio of timekeepers to nontimekeepers is 400:100, or 4:1.

8. Provide whatever statistics the firm has (e.g., from document management or time recording
systems) on the allocation of work performed on client documents between timekeepers andnon-timekeepers?
Example: our attorneys account for 80% of the keystrokes in Microsoft Word; 98% of the checkout
time of the average document is attributable to timekeepers; the median document is nevertouched by a non-timekeeper.

9. Provide whatever statistics the firm has (e.g., from document management or time recording
systems) on the allocation of work performed on client documents at various levels (e.g.,partners, associates, paralegals)?
Example: associates account for 72% of the attorney keystrokes in Microsoft Word, partners
account for 20%, paralegals account for 8%. Associates are responsible for 70% of the PDF uploaded
to the DMS, paralegals account for 27%, partners account for 3%. Paralegals are responsible for 56%
of the spreadsheets uploaded to the DMS, associates account for 32%, partners account for 12%.

Training
Reasoning: purchasing technology is one thing. Using it properly is another. The digital
native is a myth. And most enterprise-level technology is far from intuitive. Studies suggestthat it takes a 10x investment in organizational capital—training and process redesign—toreap the full benefits of every dollar invested in new technology. * This section is aimed atdetermining whether the firm makes the complementary investment in training and whether
their personnel know how to use the technology tools at their disposal.
Rubric: fortunately, the Legal Technology Assessment is a true, quantitative benchmarkthat enables apples-to-apples comparison. Beyond the numbers, there will be firms that do
almost nothing regarding training and those with robust, mandatory programs.

10. Fill out the tables with respect to the firm’s performance (COBOT level) on the identified
modules of the Legal Technology Assessment.
Word
Expert*Qualified

NoviceBeginner

Not Taken

Partners

1

2

2

5

100

Associates

5

25

100

75

65

Paralegals

2

5

7

6

0

Support Staff

8

21

31

20

0

E. Brynjolfsson and L. M. Hitt, “Computing Productivity: Firm-level Evidence,” Review of Economics and Statistics
8, no. 4 (2003): 793– 808.

ExcelExpertQualified

NoviceBeginner

Not Taken

Partners

1

0

2

3

104

Associates

2

5

81

102

80

Paralegals

0

3

7

10

0

Support Staff

4

13

23

40

0

Qualified

NoviceBeginner

Not Taken

PDF
ExpertPartners

1

1

3

5

100

Associates

4

19

83

99

65

Paralegals

2

3

8

7

0

Support Staff

9

18

23

20

0

11. What screening mechanisms does the firm use regarding basic fluency with common
technology? Identify to whom the screening applies (associates, staff, both).
Example: we use a customized Word assessments from eSkill for secretaries and word processors.

12. Describe any other mandatory technology training or technology-based competenceassessments the firm utilizes to ensure personnel have adequate skills?
Example: we have mandatory training with every new rollout. For example, we had one hour ofcompulsory training for our new document management system.

Electronic Signatures
Reasoning: electronic signatures are the lowest hanging fruit for going paperless. A
substantial percentage of printing is attributable solely to the perceived need to affix an ink
signature to a document, only for the document to be scanned back in electronic form. Thisis a waste of time and resources that degrades the quality of the document (worse
appearance, larger file size, reduced searchability). Lawyers, more than anyone, should
be expected to know that the law recognizes the validity of electronic signatures with only
a few minor exceptions (e.g., codicils, notarized documents).†
Rubric: one end of the spectrum will be the firms that still print, sign, and scan everything.
On the other end, firms will have comprehensive e-signature systems, including a complete
audit trail, integrated directly into their standard workflow.

13. What electronic signature software or protocols (including number of licenses, whereapplicable) does the firm deploy?
Example: we have created a customized digital signature for each lawyer in Word and Adobe
Acrobat. Also, each of our 68 admins have been given logon credentials to Adobe EchoSign andtrained to circulate documents to clients and opposing parties for electronic signature.

14. Is it standard practice at the firm to use electronic signatures on letters, pleadings, etc.? That is,if the median partner sends a signed letter to opposing counsel is it more likely that they(a) have it printed out, sign it, scan it, and send it or (b) sign it and convert it to a PDF
electronically.
Example: yes, it standard practice to electronically sign all documents. Our templates are set up with
document tags for electronic signatures. Most of our partners can now accomplish the electronic
execution of documents without secretarial assistance.

15. Provide whatever statistics the firm has on the utilization of electronic signatures by attorneys.
Example: an average of 414 documents are signed each month via the firm’s DocuSign portal. Thereis also an average of 11 unique uploaders and 36 unique firm-based signatories each month.

†

Flaherty, D. Casey, and Corey Lovato. "Digital Signatures and the Paperless Office." Journal of Internet Law 17.7
(2014): 3-12.

Document Assembly
Reasoning: document automation is a rationalized, systematic approach to cut-and-paste.
Reinventing the wheel is bad but so is repurposing a poorly constructed wheel. Document
automation starts with pristine templates that are iteratively improved. But the technology
can go well beyond that with dynamic questionnaires (i.e., a decision tree where thequestions change based on answers to previous questions) and automated workflows.
Rubric: some firms won’t use document automation at all. Others will use it in a limited
capacity. And some firms will have made a substantial investment in document automation,
including the training necessary to integrate it tightly into their timekeeper’s standard
workflow.

16. What document automation tools (version and number of licenses) does the firm deploy?
Example: the firm has an enterprise license for HotDocs 10.

17. Describe the processes in which document automation is utilized.
Example: all the firm’ NDA’s and pleading templates for new matters are created using a custom
document assembly macro.

18. Provide whatever statistics the firm has on the utilization of document automation.
Example: 31 unique users generate 213 documents per month through the firm’s HotDoc’s portal.

Process/Project Management
Reasoning: process is critical to ensure that (a) the right people (b) are doing the rightthings (c) the right way. Good process can enhance efficacy and efficiency. Bad process
can be a major drag on productivity. The first question is whether the firm has processes.
The second question is whether those processes are any good.
Rubric: some firms will consider process to be silly encumbrance on their autonomous
professionals. Other firms will have bought into the Lean/Six Sigma/Agile/ProjectManagement ethos of higher quality at lower cost in less time.

19. How many process maps has the firm created or updated within the last five years? How many
checklists has the firm created or updated within the last five years? In what areas does the firm
use process maps or checklists?
Example: The firm has 7 process maps for preparing patent filings. We also have a local rulechecklists for filings in the 19 courts in which the firm has active matters.

20. Describe the firm’s project management protocols and the integration of project management
into the day-to-day work performed at the firm. Describe the firm’s matter management andtracking practices from a project management perspective, including how calendars are created,maintained, and shared. Identify the software used for project management, matter tracking,calendaring, etc. as well as who is responsible for their use.
Example: Every matter budgeted for more than $100,000 is assigned a PMI-certified project
manager. The firm employs 12 full-time project managers who work directly with the relationshippartner to map out the matter at inception and then help the partner monitor the matter
throughout its lifecycle. The PM’s use a custom Onit software solution. In addition, the PM or, insmaller cases, the administrative assistant of the partner in charge is tasked with maintaining thelitigation calendar in CompuLaw, as well as a team calendar in Outlook.

21. Describe the firm’s quality assurance protocols for client documents. Indicate what softwaretools, if any, the firm utilizes for quality assurance.
Example: every timekeeper has WordRake and PerfectIt installed with Word. In addition, all of thefirm’s transactional associates and paralegals have a license for and training on EagleEye. Finally,
we have 12 junior associates designated as copy editors and available to bring fresh eyes to work
product before it goes out the door. The firm has provided these copy editors with both training
and several document-specific proofreading checklists that go beyond our automated tools.

Knowledge Management
Reasoning: one famous former general counsel often remarks, “the only thing I hate morethan answering the same question twice is paying to answer the same question twice.”
Knowledge management is not just about capturing asked-and-answered questions but
structuring the answers (conclusions, model clauses, etc.) in a manner that is easily
accessible and updateable when the questions are asked again.
Rubric: some firms will provide nothing or, at best, the ability to keyword search prior work
product. Other will firms will give billable credit for adding content to a fully-indexed,organized wiki-like repository and hold personnel accountable when they do not make therepository the first (for research) and last stop (for contributions) on their matters.

22. Describe the firm’s knowledge management solutions and protocols? How does the firm create
lasting institutional value from work performed (beyond the enhanced experience of theindividual lawyers doing the work)?
Example: the firm has a fully searchable document repository that enables lawyers to pull up
previous work product. In addition, the corporate law department maintains a clause bank that
contains 213 entries, is accessed an average of 8 times per day, and is updated an average of twice
per month.

23. Does the firm award billable credit for contributions to the institutional knowledge base? How
does the firm ensure that work product is translated into generally useful reference material?
Example: the firm does not provide billable credit for contributions to the knowledge management
system and leaves it to the individual attorneys to determine what and how they will contribute toknowledge base of the firm.

24. Provide whatever statistics the firm has on the utilization (both generation and consumption) of
its knowledge management tools.
Example: there are 42,516 documents in our knowledge management repository, which is searched
an average of 19 times per month.

Data/Analytics
Reasoning: one useful definition of information is data organized in a manner that reduces
uncertainty. Law firms have all sorts of data that, if organized properly, should help a client
reduce uncertainty about costs and outcomes. Many firms, however, are data rich but
information poor.
Rubric: one end of the spectrum will contain firms that see the value in capturing their datain order to turn it into actionable information. The other end includes those firms who think
the intuition of their experienced attorneys does not need to be augmented by empiricalanalysis.

25. How much does a motion for summary judgment cost? How many MSJs has the law firm filed in
the last two years? What has been the median cost to the client? How are the costs distributed(i.e., on a normal, bell-shaped curve vs. a power-law distribution etc.)? How granular can thefirm break down the types of MSJ’s (jurisdiction, topic, etc.)? Does the firm keep statistics on theoutcomes?
Example: the firm filed approximately 850 motions for summary judgment in the last two years. But,
without considerable labor, we cannot provide a median cost to client, let alone a distribution bycost or type. We also do not maintain records on outcomes.

26. Describe how the firm’s uses of data and analytics in budgeting and cost projection. Provide
whatever statistics are available in terms of the consistency with which the firm stays withinmatter budgets.
Example: the firm has a full-time pricing director who analyzes all of our matters and assists in
putting together budgets and bidding on fixed-fee work. She has created a dashboard that tracks
individual matters as well as aggregate budgeting information. The dashboard will be shared during
the in-person review. The firm, however, does not have any statistics on performance against initialbudget.

27. Describe the firm’s use of data and analytics to inform substantive and strategic decisions withina matter.
Example: the firm uses reports from Lex Machina to help guide tactical decisions in IP litigation.
Beyond that, however, the firm does not have any ability to mine data for making decisions at thematter or task level.

Billing Hygiene‡
Reasoning: Delayed recording of work degrades the accuracy and the usefulness of theentries. Moreover, there is evidence to suggest that the longer a timekeeper waits, themore they inflate. Even in the flat fee context, inaccurate record keeping can affect costaccounting and, as a result, matter pricing.
Rubric: one end of the spectrum will be the firms where most everyone waits until the endof the month to reconstruct what they did. On the other end, there will be the firms thattrack their matters in real time and penalize late record keeping.

28. What is the firm’s policy on how immediately timekeepers must enter their time? How is thepolicy enforced?
Example: Our policy requires timekeepers to enter their time daily. If they have not made an entry
within a week, we send them a reminder email.

29. Based on the information contained in firm’s timekeeping database, what is the average delay intime recording? That is, what is the average delta between the activity and the capturing of time
related to that activity?
Example: the average delay between work being performed and time being recorded is 10.2 days.

30. Does the firm provide real-time access to timekeeping data internally (partners) or externally(clients)? How?
Example: our partners have access to a project-management dashboard that allows them to trackmatter progress in real time. But we do not provide access to clients.

‡

The author offers a separate consulting service that goes beyond immediacy of entry. Algorithms are used toreview invoices for block billing, vague entries, skills mismatch, duplicate entries, padding, etc.

Additional
Reasoning: the Review questions are comprehensive, but they will never be complete.
Even as the questionnaire is tailored to specific situation and evolves over time, it will notcover every conceivable area that might be of value. It is only fair that firms have theopportunity to highlight their achievement and their aspirations. Moreover, for the exchange
to be a true dialogue, the firms deserve an opportunity to point out areas where the client
might improve.
Rubric: this section is not graded. But it affects the overall assessment, as well therecommendations.

31. Describe up to three projects that the firm has completed or refreshed in the last three years
that have an appreciable positive impact on the process and technology aspects of delivering
legal services but did not fit appropriately into responses to the previous questions.
Example: FirmSync is a secure Dropbox-like program that allows the firm to easily share and updatelarge volumes of data and large quantities of files with our clients. It works much better than an
extranet or FTP for sharing files with our clients.
Sekundment is a program by which the firm provides free secundment of junior associates to clientsto help them solve policy, process, or technology related issues. The associates have all been
through intensive technology and Lean Six Sigma training. They are sent to the client to tackle adiscrete policy, process, or technology-related issue. We consider it part of their training to be readyto assist on the client’s work.
The NeXt prize is a firmwide program that rewards process innovation. It has a heavy technologicalbent. 2% of the associate bonus pool is withheld to create prizes for personnel (staff is welcome toparticipate) who propose the best, most actionable process improvement. Last year’s winners
included automation of the employment litigation intake process and Outlook inbox rule that
helped ensure that client emails would receive a response within 24 hours (even if only toacknowledge receipt of the email and promise a more complete response at a later date).

32. Describe up to three projects that the firm is currently undertaking or is about to initiate thatwill have an appreciable positive impact on the process and technology aspects of delivering
legal services but did not fit appropriately into responses to the previous questions.
Example: the firm has brought in an outside vendor to complete a consolidation and refresh of all
the firm’s litigation templates. The project should be done in Q1.
The firm is using Umbria to create matter and practice management dashboards that update in realtime and are available to all our partners. The prototype will be complete in Q2. We anticipate
rollout in Q1 of next year.

33. Identify areas in which process or technology upgrades on the part of the client could facilitate
the firm’s delivery of legal services.
Example: the client still has not upgraded from Office 2003. Not being able to send the client .docx
and .xlsx files is an impediment to sharing work product.
The client also has to approve every individual timekeeper in its matter management system at thecommencement of every matter. The system automatically rejects timekeepers whose bills predatetheir approval in the system. But the client is often delayed in setting up matters in the system, letalone approving timekeepers. It can be weeks before a litigation-initiating pleading transmitted via
email turns into a formal matter in the system. It can be several more weeks before timekeepers
are approved. The client’s rules are at odds with the client’s own practices, demanding that games
be played to get the work going in a timely matter.

Onsite Review Protocol
Reasoning: responses to the questions can only provide so much insight. An onsite review
adds considerable perspective. How the sausage gets made may appear very different
from the vantage point of (a) firm management, (b) delegators, and (c) delegates. For
example, while a firm may claim to use electronic signatures, the fact that e-signatures are
not utilized for something as routine as an engagement letter suggests otherwise. Having
purchased project management software is not the same as integrating its use into a welldesigned workflow. The onsite review facilities a much deeper dive into the firm’s process
and technology. It also gives them the opportunity to show off.
Rubric: this section is not graded. But it affects the overall assessment, as well therecommendations.

The interview requests will be modified based on the firm’s answer to the foregoing. But, initially, theinterviewer will want to meet the individuals responsible for putting together those answers. Theinterviewer will also want to meet, at their workstation, with the individuals responsible for generating:Engagement Letters

Motions/Exhibits

Matter-Specific Analytics

Pleadings

Settlement Agreements

Billing Analytics

Litigation Calendar

Case Files

Discovery Requests/Reponses

Matter Budget

Deposition Questions/Exhibits

Checklists/Process Maps

Legal Research/Memos

Knowledge Management

There may be more than one individual involved in each process. Engagement letters, for example, may
be a collaboration between a partner and her secretary. Or discovery responses may be a collaboration
between an associate and word processing. The interviewer will meet with each party involved. Copies
of related process maps and checklists are much appreciated.

[The above is just an example for general commercial litigation. It will be customized based on thespecific type of work the firm performs for the client (e.g., patent, real estate, class actions,employment, transactional)]

Basic Technology Benchmarking
Read about the LTA in The Washington Post and the ABA Journal. Understand the positive case for technology training.
Watch an introductory video here.

Basic Technology Benchmarking utilizes the Legal Technology Assessment to establish how fluent legal
professionals are with basic technology, such as word processing and spreadsheets. The LTA is both abenchmarked assessment and a training platform. Benchmarked results can be used for marketing,professional development, team assembly, onboarding, rate negotiation, invoice review, provider
differentiation, etc. Primarily, however, results should be used to ensure that legal professionals are
getting the training they need. The LTA pairs competence-based assessments with synchronous, active
learning in order to provide an effective, tailored training solution. The objective of the initial version of
the LTA is its own obsolescence in the near term.
The Basics
The LTA is taken in a completely live environment on exemplar documents. The LTA is digitally-signed
Word plug-in that can be downloaded and installed on the user’s own computer or on a group computer
provided by their organization (e.g., training laptop, computer lab).
The instructions are contained in a Word document with a control panel on the side. A screenshot ishere. As shown in the screenshot, the software automatically opens and closes exemplar document on
which work is completed. On the first task, for example, the instructions to (i) turn off track changes and(ii) delete comments are presented in the instructions document while the software automatically
opens another Word document in which to turn off track changes and delete comments.
Each user has a unique username (their email) and password that enables individual score tracking. At
the completion of each module, the user is sent their score. An introduction to the individual score
report is here. The score sheet identifies specific deficiencies and enables the creation of individually
tailored training curriculum. After training, the user can retake the LTA as many times as they wantduring the license period (usually one year).
There is also a Training Edition that pairs the assessment with synchronized learning. Instead ofreceiving a score at the end of the module, the user is informed at the completion of each task whether
they performed the task correctly. As shown in the screenshot, the control panel of the Training Edition
has not only a Previous Task button, which allows the user to navigate backwards to retry the previoustask, but also a Help button. The Help button is programmable. It can open up any training material
designated by the organization. For organizations that lack internal or external training resources,
Hotshot has created professional videos that walk the user through each task. The user can try, fail,return, learn, try, and succeed in a synchronous loop.
The Training Edition is not connected to any database. Performance is not recorded. Users can use theTraining Edition for self-evaluation and preparation in a low-stakes setting. Embarrassment is not theobjective. Getting legal professionals properly trained is the objective. Even recorded scores are notreleased without the express permission of the organization. Indeed, the bare fact that individuals in theorganization have taken the LTA is not disclosed absent express permission. That said, law firms andtheir clients will take an interest in LTA scores. A walkthrough of benchmark reporting is here.
20

The LTA is modular. Not only is the assessment broken into separate modules by software application,
but the features assessed are customizable. A list of features for which assessments and training are
currently available follows.
Word













Accept/Turn-off changes and comments
Cut & Paste
Replace text
Format text
Footers
Insert hyperlink
Apply/Modify styleInsert/Update cross-references
Insert page breakInsert non-breaking space
Clean document properties
Create comparison document

Excel













Copy/Rename worksheet
Insert column
Format column width
Format text
Sort
FilterRemove duplicates
DivideCount
Sum
Average
Prepare to print

PDF












Convert Word & Excel documents to PDF
Create single PDF from multiple files
Recognize text (OCR)
Extract page
Highlight text
Redact information
Insert footer
Create bookmark
Create internal link
Remove hidden Info
Password protect

21

The LTA Advantage
By pairing competence-based assessments with synchronous learning the LTA augments traditional
training in a number of ways.


Sorting mechanism. Not everyone is a poor end user of common software. Some have training.
Others are self-taught. In onboarding, the LTA can distinguish between previously trained anduntrained users. On a go-forward basis, the LTA segments trainees by skill set, which includes
letting skilled user test out of training they do not need.

Lose your illusion. Most computer users don’t know what they don’t know. They are notintentionally avoiding the quality-enhancing, labor-saving features available in common
software. They are simply not aware these features exist, let alone how to use them. They
believe that the way they are operating is right (or close to right). They are wrong. But they
don’t know they are wrong. Consequently, they do not recognize the need for training. Simply
mandating training will not change this preconception, and the users will be disengaged fromthe outset. The LTA can puncture their delusions of adequacy and help them realize there is aproblem in need of correction.



Tailored training. One of the easiest ways to lose a training audience is to start by teachingthem things they already know. Yet, most training classes begin with the basics for good reason:some audience members are likely to be unfamiliar with the basics. Meanwhile, everyone else
checks out. This shotgun approach to training reduces engagement. Not only does the LTA
demonstrate that training is needed, but it also pinpoints what training is needed. The LTA
identifies specific deficiencies and enables the creation of individually-tailored training
programs. It even offers a way to deliver that training in the form of the Training Edition.



Active learning. Answers can be memorized. Skills must be learned. When a child has a swim
test, knowing they will have to traverse the length of the pool unassisted is of little assistance inaccomplishing that feat. Rather than abstract knowledge, deliberate practice is necessary topass the test. We help them across the pool as many times as is needed until they can do it
their own. The Training Edition outfitted with a Help button (teach me how to do this now)
allow trainees to try, fail, study, try again, and succeed in a synchronous loop. They can identify
weaknesses and address those weaknesses in the same exercise until they are ready for the live
exam.



Validation. With traditional tech training, all we know is that the trainees sat in a room orwatched a video for a prescribed amount of time. Whether they learned anything is a matter of
speculation. Time is a poor proxy for learning. Competence-based assessments both establish abaseline and measure progress against that baseline.

Self-direction. Different people learn different ways and will struggle to different degrees.
Some thrive with a live teacher. Others are too embarrassed to ask questions. As long as
sufficient training resources of varying modalities are made available, the validation componentof LTA allows trainees to self-direct their remedial efforts and substantiate how effective thoseefforts were.
22

The Future
Our modest goal is to replace the virtually meaningless “proficient in Microsoft Office” with COBOT
Qualified in specific software. We intend to expand our offerings both horizontally (other programs suchas Outlook and PowerPoint) and vertically (different levels from Delegator to Expert). Our vision for the
future involves getting the entire legal community to care about, and take an active role in, technology
training.


Law schools. Law schools are creating more and more classes that are directly related totechnology (e-discovery). This is a good thing. But basic technology training does not require
separate credits. Basic technology competence can be integrated directly into the corecurriculum. In Civil Procedure, students learn how to prepare an e-filing (PDF). In Contracts,
they learn how to format a contract (Word). In Corporations, they learn how to handle financial
data (Excel). Etc. All of these modules can be provided outside of the classroom through aplatform like ours without the instructor needing to alter anything other than their syllabus.
Additional integration points include pre-enrollment learning, legal writing, and pre-clinic
bootcamps.

State bars. State bars can assist members in fulfilling their evolving ethical duty of competence
by ensuring that CLE credit is available for technology skills acquisition—i.e., learning to properly
use the tools of the profession. State bars could also move CLE into a more active-learning
direction by offering CLE credit for demonstrations of competence instead of time. Currently,
time is used as a proxy for learning. All we know that the learner sat in a room (or turned on avideo) in which information was shared. We do not know whether they absorbed any of theinformation. They just as easily could have spent the time responding to client emails or playing
Angry Birds. Credit seekers should at least have the option of a competence-based assessmentthat establishes their engagement with the material. Eventually, bars might also consider making
demonstrations of competence mandatory for newly-minted lawyers.

Legal Employers. Law departments, government agencies, law firms, alternative service
providers, and nonprofits should encourage their legal professionals to expand theircompetencies to include the use of basic technology. The level of training required will vary
based on the type of work the individual performs. Legal employers should make basic techcompetence part of their onboarding, training, and professional development protocols. They,
however, should not bear all the burden. As discussed above, law schools should produce
properly trained personnel, and state bars should support remedial efforts by offering CLE
credit.



Corporate Clients. Corporate clients should take a keener interest in how legal services aredelivered. Just as corporate clients are unlikely to use a lawyer who are unable to communicate
via email, corporate clients should have limited tolerance for a poor use of basic officetechnology. Corporate clients should use validation and benchmarking methods like the LTA tohave a structured dialogue with their outside providers about priorities and the proper deliveryof legal services.

23

LTA Instruction Sample

24

Individual Score Reporting
Every time that an individual takes a module, they receive a score report. Below is an example score report for
Assignment I (Word Processing).

Performed Correctly. Whether or not the user completed the task correctly. In the example above, the person did notproperly complete Task #6.
Target Time. Approximation of how long the task should take a qualified user. Target time is double an expert timerounded up with a minimum of one minute despite the fact that many of the tasks can be completed in a matter of
seconds.
Actual Time. How long a task actually took the user.
Penalty Time. Approximation of how long the task would take a beginner. The penalty time is substituted for (not addedto) the actual time when the task is performed incorrectly. The theory is that the individual (or their firm) will reach theright result eventually, it will just require more time.
Assessed Time. Equals the actual time if the task is performed correctly. Equals the penalty time if the task is not
performed correctly. All assessed times that exceed the target times, even those where the individual performed thetask correctly, are highlighted in red. This is a quick, visual cue to indicate where the user needs to improve. The
individual score report is not only a record of how well the user performed, it is also a mechanism to create anindividually-tailored learning plan.
The next page contains a sample score report for all three modules.

25

In addition to the individual assignment reports, the assignments scores are translated into billable time and presented
in summary format.

26

Finally, each individual is given a COBOT designation for each module.

COBOT: Certified Operator of Basic Office Technology. An individual who hits the target time is Qualified. Since thetarget times are deliberately generous, there is also Expert status at 50% of the target time. The cutoff for Novice is 3x
the target time. Anything beyond 3x the target time represents Beginner level skills.
In concrete terms, an Expert will complete an individual module in 0.2, or less. To be designated Qualified, the user willneed to achieve times of 0.4 per module. The cutoff for Novice is 1.2 per module, or 3.6 for the entire LTA. That is, theNovice may spend as long on a single module as the Qualified user spends on the entire LTA. Anything worse thanNovice is a pure Beginner. The table of designations is below.
ExpertQualified

NoviceBeginner

Word Processing

≤ 0.2

≤ 0.4

≤ 1.2

>1.2

Spreadsheets

≤ 0.2

≤ 0.4

≤ 1.2

>1.2

PDFs

≤ 0.2

≤ 0.4

≤ 1.2

>1.2

Importantly, and as will be discussed in more detail in the Benchmark Reporting section, the individual scores can beweighted so that a user who does not spend time with a particular type of document (e.g., Excel spreadsheets) is not
penalized for a poor performance on the related module. Indeed, since the weighting can be set at 0%, the user can skip
module entirely without negative impact on the overall score reported.
Individual users may take the LTA as many times as they need to. The first attempt should not be about getting a
Qualified score. Rather, the first attempt should be aimed at identifying knowledge gaps and creating an individualized
training program. Subsequent attempts serve to validate the efficacy of that training. With respect to Benchmark
Reporting, we only release each user’s most recent score and then only after we are given express permission to do so.
Clients only see scores that are released and otherwise are not able to differentiate between firms that have used the
LTA for self-assessment and firms that have not.
The biggest gains come from identifying and training Beginners. And there are a lot of efficiencies in turning a Novice
into a Qualified user. It would be fantastic if everyone were Expert in all aspects of basic, ubiquitous software. But thatgoal might prove impractical. A more reasonable objective might be to have a few Experts in each area available so thatthey can handle the larger, software-intensive projects and be there to assist their colleagues on the smaller ones.
Individual score sheets will be aggregated for both Group Score Reporting and Benchmark Reporting.

27

Group Score Reporting
Individual score reports will be aggregated in whatever manner and provided at whatever interval the organization
selects. The goal is to keep the organization apprised of progress while also providing their training staff the detailed
information they need to properly upskill the individuals subject to the LTA.
Group score reporting starts with bulk provision of the individual score reports.

Individual
score reports

28

Individual scores are then aggregated into tables to provide a general overview of where the initiative stands.

29

In addition to the standard tables, the organization is provided raw data down to the individual task level.

30

The raw data can then be filtered in multiples. The organization may, for example, want to zero in on which specifictasks an individual does not perform correctly:

Or the organization might want to identify all the individuals who are unable to perform a specific task:

The raw data can also be used to create pivot tables that allow the organization to dissect the information in whatevermanner they want. For example, before determining who struggled with which task, the organization may need to
identify the tasks on which the most users struggled:

But these are just examples. The raw data can be manipulated any number of ways. And, again, the standard reports will
be provided in whatever form and at whatever interval the organization requests.

31

Benchmark Reporting
In the following exemplars, all names have been changed to protect the innocent and guilty alike. To the extent any of
the names are those of an actual legal professional, it simply an awesome coincidence that their name appears in The
Song of Fire and Ice (a.k.a., Game of Thrones), from which all names are taken. The firm names are also fictitious—simply
a jumble of firm names from the AmLaw 200—and have zero relationship to the firms actually represented.
When released to clients, scores are reported in composite form so that the clients are not only able to benchmark
individuals versus the target time but also benchmark their firms against each other.
Raw Scores are provided on the individual level:

Waste %: The percentage of overall time that is above the target. With a target of 1.2 hours, an individual who takes 1.4
hours has a Waste % of 14% because 0.2 hours of their 1.4 hours were avoidable waste (i.e., 0.2/1.4 = 14%). Likewise, ascore of 1.6 hours is a Waste % of 25% (because 0.4 hours of the 1.6 hours was waste, or .4/1.6 = 25%) and a score of 2.4
hours is a Waste % of 50% (because 1.2 hours of the 2.4 hours was waste, or 1.2/2.4 = 50%).

32

COBOT: Certified Operator of Basic Office Technology. An individual who hits the target time is Qualified. Since thetarget times are deliberately generous, there is also Expert status at 50% of the target time. The cutoff for Novice is 3x
the target time. Anything above 3x the target time represents Beginner level skills.
Of course, legal professionals do not spend all of their time on the type of tasks assessed on the LTA. Indeed, some legal
professionals should be exempted entirely because they do not perform these types of tasks. For everyone else, clients
can—with input from their firms—populate the results spreadsheet with Assumptions about how different timekeepers
allocate their average day.

The default presumption is that equity partners are not doing this kind of work. The default assumptions are thatparalegals spend a greater percentage of their time on these types of tasks than associates, who spend a greater
percentage than service partners. But these are just general assumptions that can be modified at the individual level—i.e., changed for each individual user reported.

33

Law firms should have data from their document management system to provide a rough picture of who spends theirtime doing what. Thus, for example, if a specific legal professional never uses Excel, their Excel percentage can be
reduced to 0% and their performance on the Excel module (which they can skip without penalty) will have no impact on
their Recommended Writedown. Recommended Writedown is a composite result that combines the user’s raw scorewith the adjustable assumptions about how their time is spent.

The concept of Recommend Writedown may seem harsh at first glance. But the entire object of the LTA is to avoid thesetough conversations entirely by creating incentives for proper training. The LTA is not intended to embarrass. But,
unfortunately, the LTA results thus far suggest that the prospect of embarrassment is necessary to incent legal
professionals to take technology training seriously. Since they have access to the Training Edition and can retake the LTA
as many times as they like—we only report the most recent score—every legal professional subject to the LTA shouldhave ample opportunity to achieve a qualifying score.

34

Individual scores are rolled up and benchmarked at the firm level in a variety of ways:

35

We only release the most recent score and only after the firm authorizes us to do so. If the firm gets a score requestfrom a client, they can pass it on to us with express permission to release the scores identified. Alternatively, if we
receive a score request from a client, we forward it to a designated person at the subject firm and then release only
those scores, if any, that the firm authorizes. In fact, we do not even identify whether a firm or an individual has a scorein the system unless we are expressly authorized to do so.
Thus, a client request may include two firms that will not release their scores for different reasons: one because they
don’t have scores; the other because they are not ready to release their scores. From the client perspective, the two
firms will look exactly the same. We will simply indicate that the firms declined to release their scores without specifying
whether or not they have scores in the system. We do not want a firm that is in the process of evaluating their internal
competencies to be at a disadvantage vis-à-vis a firm that is not inclined to do so.
Obviously, clients can apply pressure to get firms to take the LTA and release their scores. In addition, our score
reporting to clients provides a method to impute scores to Decliners—i.e., firms or individuals whose scores are notreleased.

For more information go to: www.procertas.com

36