Patent ID: 12242545

DETAILED DESCRIPTION

The Figures (FIGS.) and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. A letter after a reference numeral, such as “120A,” indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as “120,” refers to any or all of the elements in the figures bearing that reference numeral.

The figures depict embodiments of the disclosed system (or method) for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

Overview

A document management system can assist users in identifying upcoming expiring agreements. This can be helpful for example if a client has multiple agreements that will require renewal negotiations in the near future. The document management system uses a trained negotiation prediction model to predict how long it is likely to take for agreements to be re-negotiated. Based on the prediction, the document management system may prompt a user of a client device to begin the renegotiation process at least a predicted interval of time ahead of the expiration date of the agreement. Although examples herein describe agreement re-negotiations, this process may be used in other situations, such as for documents with associated filing deadlines, or for documents which are regularly generated and updated or that are time-sensitive, such as monthly calendars, and bi-monthly supply request forms.

The document management system also monitors standard clauses included by a user in documents and, when a clause is updated, flags other documents that include the same clause for updating. In this way, a user can be notified of other documents that the user may also want to update to include the language of the updated clause. If the difference between the instance of each clause in a flagged document and the updated clause language exceeds a threshold value, the documents may be presented to the user within an interface that enables the user to automate the process of updating the language of the outdated clause instances in the documents.

The document management system additionally uses machine learning models to identify a type of document with which a user is interacting and to suggest actions that can be taken on the document. Examples of actions that may be taken with respect to a document include replacing text with fields, replacing clauses with pre-approved versions of the clauses, and synchronizing the document with a third-party system. Action options may be based on actions taken by similar users, actions done to similar documents, and actions taken in similar contexts (e.g., at similar times of the year). In one embodiment, the document management system presents the recommended actions within an interface that enables a user to view and select desired actions from a list of suggested actions and that enables the user to automate the performance of the actions if requested

System Environment

The system environment described herein can be implemented within an online document system, a document execution system, or any type of digital transaction management platform. It should be noted that although description may be limited in certain contexts to a particular environment, this is for the purposes of simplicity only, and in practice the principles described herein can apply more broadly to the context of any digital transaction management platform. Examples can include but are not limited to online signature systems, online document creation and management systems, collaborative document and workspace systems, online workflow management systems, multi-party communication and interaction platforms, social networking systems, marketplace and financial transaction management systems, or any suitable digital transaction management platform.

FIG.1is a high-level block diagram of a system environment for a document management system, in accordance with an example embodiment. The system environment enables client systems associated with a document management system140to create and manage digital documents, e.g., for electronic execution to one or more receiving parties. A client may review, modify, and execute generated documents. As illustrated inFIG.1, the system environment includes documents for analysis110, one or more client devices120(e.g., client device120A and client device120B), a document corpus130, and a document management system140, each communicatively interconnected via a network180. In some embodiments, the system environment includes components other than those described herein. For clarity, although two client devices120are shown inFIG.1, alternate embodiments of the system environment can have any number of client devices120. For the purposes of concision, the web servers, data centers, and other components associated with an online system environment are not shown inFIG.1.

The documents for analysis110are analyzed to identify upcoming expiration dates of stored documents for renegotiation purposes, to identify changes to clause language and portions of other documents with similar clause language that may need to be updated, and to identify document types for action recommendations based on actions taken with respect to similar documents and/or similar users. Examples of documents for analysis110include but are not limited to: a sales contract, a permission slip, a rental or lease agreement, a liability waiver, a financial document, an investment term sheet, a purchase order, an employment agreement, a mortgage application, and so on. Each document for analysis110can include one or more clauses. Clauses are portions of a document, and may include text, images, or other material of any length. In some embodiments, a clause corresponds to a legal clause, a business clause, financial agreement text, and the like. A given document in the documents for analysis110may include multiple clauses that each correspond to a different header within the document. Clauses may further be associated with one or more clause types that characterize content of the clause as corresponding to a particular category of the clause. Examples of clause types include but are not limited to an indemnity clause, a merger and integration clause, a severability clause, a fees clause, a damages clause, a pricing clause, a purchase clause, a payment clause, and so on. It should be noted that not every portion of a document or of text within a document may be considered a “clause” for the purposes of the description here.

A client device120, such as client device120A, provides the set of documents for analysis110to the document management system140(or provides instructions to create documents with information from the client device120). Although examples are given herein in the context of a set of documents for analysis110, the document management system140can coordinate the creation, viewing, editing, analyzing, and signing of any number of documents (e.g., thousands, millions, and more) for any number of users or accounts, and for any number of entities or organizations. The client device120is a computing device capable of transmitting or receiving data over the network180. The client device120may be a conventional computer (e.g., a laptop or a desktop computer), a server, a cell phone, or a similar device. The client device120enables a user to create a set of documents for analysis110and to provide the set of documents for analysis110to the document management system140. After the document management system140analyzes or otherwise makes use of the set of documents for analysis110, the client device120may provide information to a user of the client device120describing any results related to the analysis (e.g., via display of a user interface). In some embodiments, the client device120receives an interface for display from the document management system140.

The document corpus130includes document information for the document management system140. The document corpus130may include a set of training documents (e.g., for use by the document management system140in training machine learning models). Documents in the document corpus130may be stored in association with document metadata. Document metadata may include training labels for training certain machine learning models and may additionally include various information describing documents or clauses of documents (e.g., clause metadata), such as user-provided classifications for clauses. In some embodiments, the document management system140may analyze or otherwise interpret the set of documents for analysis110in isolation or with reference to a particular subset of documents of the document corpus130. The document corpus130may include documents provided by one or more parties, such as a party associated with the client device120. In various embodiments, the document corpus130may be stored locally on the document management system140or may be stored remotely from the document management system140(as shown inFIG.1).

The document management system140is a computer system (or group of computer systems) for storing and managing documents for various entities. The document management system140receives information from components of the system environment including the documents for analysis110and the document corpus130. The document management system140uses this information to determine information related to the documents for analysis, such as deadlines associated with upcoming agreement re-negotiations, updates to clauses in similar documents, and predictions of certain actions that a client may want to take with respect to a document. In some cases, the document management system140may coordinate a collaboration process between a party and a counterparty, for example, between clients associated with client device120A and clients associated with client device120B.

The document management system140can be a server, server group or cluster (including remote servers), or another suitable computing device or system of devices. In some implementations, the document management system140can communicate with user devices (e.g., the client device120) over the network180to receive and provide information (e.g., documents or document analyses).

The network180transmits data within the system environment. The network180may be a local area or wide area network using wireless or wired communication systems, such as the Internet. In some embodiments, the network180transmits data over a single connection (e.g., a data component of a cellular signal, or Wi-Fi, among others), or over multiple connections. The network180may include encryption capabilities to ensure the security of customer data. For example, encryption technologies may include secure sockets layers (SSL), transport layer security (TLS), virtual private networks (VPNs), and Internet Protocol security (IP sec), among others.

FIG.2is a high-level block diagram of a system architecture of a document management system, in accordance with an example embodiment. The document management system140includes various modules and data stores to determine upcoming agreement re-negotiation deadlines, suggest clause updates to related or similar documents, and to suggest actions a client may want to take with respect to a document. The document management system140includes a database205, a model generator210, a model store215, a document generator220, a document review module225, an expiration identifier, a time prediction module235, a collaboration module240, an edit detector245, a document comparison module250, a document type predictor255, an action store260, and an action identifier265. InFIG.2, the modules have been grouped according to system processes that they support for clarity. The expiration identifier230, the time prediction module235, and the collaboration module240are all used by the document management system140for predicting re-negotiation intervals. The edit detector245and the document comparison module250are used by the document management system140for updating similar clauses across documents. The document type predictor255, the action store260, and the action identifier265are all used by the document management system140for suggesting document actions to a client. Computer components such as web servers, network interfaces, security functions, load balancers, failover servers, management and network operations consoles, and the like may not be shown so as to not obscure the details of the system architecture. Additionally, the document management system140may contain more, fewer, or different components than those shown inFIG.2and the functionality of the components as described herein may be distributed differently from the description herein.

The database205stores information relevant to the document management system140. The stored data may include, but is not limited to, the set of documents for analysis110, clauses within the set of documents for analysis110, the document corpus130, clusters of document clauses, and so on. In some embodiments, the database205stores metadata information associated with documents or clauses, such as documents labeled with training data for machine learning models. The document management system140can update information stored in database205as new information is received, such as new documents for analysis, results of analyses performed by the expiration identifier225, the edit detector230, the time prediction module235, the document type predictor240, and the document comparison module245. The document management system140can also update information stored in the database205based on user input provided via user interfaces as may be generated by the document review module260.

The model generator210trains machine learning models that are used by various modules of the document management system140. In various embodiments, the model generator210may use different versions of supervised or unsupervised training, or another training technique to generate and update the models. The models may be any appropriate machine learning model, including, but not limited to, decision trees, regression algorithms, support vector machines, and neural networks. Two models that may be generated and maintained by the model generator210include a negotiation prediction model325and a document type prediction model825. Additional details about these specific models are provided in reference toFIG.3andFIG.8. To train the models, the model generator210may access training data, such as labeled documents stored in the database205. For example, training data for models may come from the document corpus130. The model generator210trains the models using the training data and stores trained machine learning models in the model store215. In some embodiments, the model generator retrains the models stored in the model store215periodically, or as new training data is received.

The model store215stores machine learning models for the document management system140. In some embodiments, the model store215may store various versions of models as they are updated over time. In the example herein, the model store215stores a negotiation prediction model325and a document type prediction model825. In some cases, the model store215may store multiple versions of a type of model, for example, to apply to different document types or to other variations of available inputs.

The document generator220manages generation and modification of documents. In some cases, a document may be generated at a client device120and then sent to the document management system140for storage or analysis. In some cases, a document may originate at the document generator220. The document generator can open new documents for viewing, editing, or other document interactions. Documents may be started based on templates stored in the database205or based on prior documents stored in the database205. In some embodiments, blank documents are also generated when needed. In addition to generating new documents, the document generator220facilitates document edits and updates. Documents may be edited in various ways. In one embodiment, updated documents are received from a client device120, and may be saved to the database205by the document generator220as a new version of the document. In another embodiment, a user of a client device120can access an interface of the document management system140and can make edits to a document via the interface. The document generator saves changes to the database205. In some cases, the document generator220can also edit documents without direct input from a client device120, such as by making updates to documents based on client instructions to update similar clauses across all documents, or such as by performing a scheduled document edit. In some embodiments, the document generator220also manages document deletion.

The document review module225facilitates human review of documents and document information. In various embodiments, the document review module160provides one or more user interfaces to client systems associated with the document management system140for reviewing documents and analysis of documents. For example, the document review module225may transmit user interfaces for rendering by a client device120to present a document or information about documents to a user of the client device120. Document information presented in the user interfaces may include document metadata, predictions about negotiations associated with a document, or actions that a user may want to take with respect to a document. The document review module225may further use user input received via the provided user interfaces to make changes to one or more documents stored in the document store205. Examples of user interfaces that may be provided by the document review module160are described in greater detail below with reference toFIG.5,FIG.7, andFIG.9.

Predicting Re-Negotiation Intervals

The document management system140can assist users in identifying upcoming expiring agreements. This can be helpful for example if a client has multiple agreements that will require renewal negotiations in the near future. An agreement document can include one or more of a contract, an employment agreement, a purchase agreement, a services agreement, or a financial agreement. The document management system140uses a trained negotiation prediction model to predict how long it is likely to take for agreements to be re-negotiated. Based on the prediction, the document management system140may prompt a user of a client device120to begin the renegotiation process at least a predicted interval of time ahead of the expiration date of the agreement. Although examples herein describe agreement re-negotiations, this process may be used in other situations, such as for documents with associated filing deadlines, or for documents which are regularly generated and updated or that are time-sensitive, such as monthly calendars, or bi-monthly supply request forms. Referring back toFIG.2, a system for predicting re-negotiation intervals includes the expiration identifier230, the time prediction module235, and the collaboration module240.

The expiration identifier230tracks information about documents with expiration dates. Documents stored in the database205can be stored in association with metadata that indicates expiration dates. In one embodiment, an expiration date associated with an agreement document is a date after which the terms of the agreement document are no longer valid. For example, a document that represents an agreement between a party and a counterparty may include a date after which the agreement will no longer be in force. In some embodiments, an expiration date may be explicitly indicated in the document metadata when the document is saved to the database205. In alternate embodiments, the expiration identifier230may use a trained machine learning model or another technique to identify expiration dates associated with documents. In one embodiment, the expiration identifier determines that all documents that have a likelihood of being associated with an expiration date have associated expiration date metadata stored. For example, the expiration identifier230in some cases may detect an agreement document without expiration date metadata and may request additional information about whether the document has an expiration date from a user of the client device associated with the document. In some embodiments, the expiration identifier may periodically review the dates associated with stored documents and may flag the documents as having upcoming expiration dates.

The time prediction module235applies trained machine learning models to predict negotiation times for documents with expiration dates. In one embodiment, the time prediction module235uses a negotiation prediction model325, which takes an agreement document as input and outputs a predicted amount of time that it will take for the agreement to be re-negotiated. Additional information about the training and application of the negotiation prediction model325is included in reference toFIG.3below. The prediction produced by the time prediction module235may be stored in the database205as metadata associated with the agreement document. In some cases, the time prediction module235may analyze a document upon receipt of the document at the document management system140. In some embodiments, the time prediction module235may update time predictions for documents periodically, or as additional training data is received to update the negotiation prediction model325. The document review module225can provide a notification of upcoming document expiration to a user at a client device120. Using the information about the predicted amount of time that an agreement re-negotiation will take for an agreement document, the document review module225can provide the notifications about upcoming document expirations such that a user of the client device120has enough time ahead of the expiration date to re-negotiate the agreement before it expires.

The collaboration module240facilitates a collaboration process between a first party associated with an agreement document and one or more counterparties to the agreement. In various embodiments, the collaboration module240works in conjunction with the document generator220and the document review module225to generate a new agreement document for updating and re-negotiating the agreement and to send notifications and interfaces to the client devices120associated with the parties and counterparties. The collaboration module240may track the progress of the re-negotiation of the agreement and may modify accounts of a user and any counterparties to enable access by the users and counterparties to updated agreement documents. For example, the client device120B, shown inFIG.1may be a client device associated with a counterparty to an agreement in a document owned by the client associated with client device120A. When the collaboration module240begins the re-negotiation process, the document generator220may produce a new agreement document for review by the client at client device120A. When the document is ready, the document review module225may send a notification to the counterparty at client device120B to request a review of the document and any additional information or signatures. In some cases, the collaboration module240may keep track of what additional information is needed to complete an agreement renegotiation by the expiration deadline. When an agreement re-negotiation is completed by the parties and counterparties, the collaboration module240stores the new agreement documents and associated expiration dates in the database205.

FIG.3is a flow diagram illustrating training and application processes for a negotiation prediction model325, in accordance with an example embodiment. The negotiation prediction model325takes, as input, metadata and other information related to a document to determine a predicted time it will take to negotiate an agreement document. The negotiation prediction model325is trained to predict how long a negotiation of an agreement is likely to take based on historical information about the parties involved in the agreement, the type of agreement, and the contents of the document.

The model generator210trains the negotiation prediction model325using a set of training data300. In one embodiment, the model generator210accesses a set of historical agreement documents used for training, determines how long each of the historical agreement documents took to negotiate, and trains the negotiation prediction model325to correlate a length of negotiation for each historical agreement document with a type of each historical agreement document. The training data300includes historical agreement documents305(i.e., past agreement documents that have been processed by the document management system140and associated metadata), historical agreement document types310(e.g., sales contract, indemnity agreement, equipment lease, employment agreement, non-disclosure agreement, etc.), and historical agreement negotiation times315(e.g., one week, two months, twenty days, etc.). In various embodiments, the training data may include additional or different training inputs, not shown inFIG.3. For example, additional training data300may include context information about when an agreement was decided and information about the parties involved in past agreements.

The historical agreement documents305, historical agreement document types310, and historical agreement negotiation times315may be provided via client devices120to the document management system140. In other embodiments, the document management system140may automatically collect the historical agreement document305, the historical agreement document types310, and the historical agreement negotiation times315to add to the training data300. In other embodiments, a user of a client device120associated with the document management system140(e.g., a system administrator) may manually input or curate a subset of the training data300. It should be noted that althoughFIG.3shows a use of historical documents and historical document information, the training data300may also include sample documents generated and labeled for the purpose of training the negotiation prediction model325in addition to historical document data.

Historical agreement documents305used as a subset of the training data300may include the text of historical documents and metadata associated with the documents. Historical agreement documents305may include documents associated with the user of the client device120, documents associated with an entity or with another user associated with the user of the client device120, and historical agreement documents may include documents associated with users having one or more characteristics in common with the user. In addition to the text and input fields in a document, the metadata stored in association with the historical agreement documents305may include document file type, file size, languages within the document, region in which the document originated, characteristics associated with the sending and receiving party of the document (e.g., size, industry, location of headquarters, revenue, corporate structure), types or categories of information or passages within the document, and the like.

The historical agreement document types310include annotations of the historical agreement document305that indicate the type of agreement or negotiation that the document represents. An agreement document type may include any document with an agreement negotiated between at least two parties, such as a sales contract, a permission slip, a rental or lease agreement, a liability waiver, a financial document, an investment term sheet, a purchase order, an employment agreement, a mortgage application, an indemnity agreement, an equipment lease, and a non-disclosure agreement.

The historical agreement negotiation times315include annotations of the historical agreement documents305that indicate the length of time it took for the agreement represented by the document to be reached. In some cases, the training information about the historical agreement negotiation times may be input manually by a user of a client device120associated with the document or by a system administrator. In some cases, the document management system140may detect an amount of time it took for an agreement to be negotiated (e.g., the time between document creation and execution of the document by all parties).

The model generator210uses supervised or unsupervised machine learning to train the negotiation prediction models325using the training data300. Different machine learning techniques may be used in various embodiments, such as linear support vector machines (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps. The training of the negotiation prediction model325helps the model to identify relationships between the historical agreement documents305, the historical agreement document types310, and the historical negotiation times315. In other words, training a negotiation prediction model325enables the negotiation prediction model325to identify a predicted time330that it will take for an agreement to be re-negotiated, given the original agreement document320and history of similar agreement documents. In some embodiments, the model generator210trains multiple negotiation prediction models325, such as a separate negotiation prediction model325for each agreement type.

A trained negotiation prediction model325can be applied by the time prediction module235to an agreement document320. In some embodiments, the negotiation prediction model325also accepts metadata associated with the document (e.g., document agreement type, agreement parties) as input. The negotiation prediction model325generates a predicted time330that it will take for the agreement in the current agreement document to be renegotiated. The document review module225may display the agreement expiration date from the expiration identifier230and the predicted time330for re-negotiation of the agreement for a client associated with the document to view in an interface at the client device120.

FIG.4illustrates an interface400of a document analytics dashboard for reviewing upcoming expiration dates of agreement documents, in accordance with an example embodiment. In the embodiment shown, the interface400includes a document list410, notification icons420, an agreement status interface430, and a re-negotiation widget440. The example interface400shows a document list410of agreement documents. In the example ofFIG.4, a user may also search for specific documents in the list using a search interface. The document list includes notification icons420next to agreements with identified expiration dates that are sooner than a threshold length of time from the present date. For example, AgreementB.docx, AgreementF.docx, and AgreementG.docx have notification icons420indicating impeding expiration dates. The interface400also includes an agreement status interface430that displays additional details about the expiration dates of agreement documents in the document list410. In the example ofFIG.4, AgreementF.docx is selected by the user, as indicated by the dotted fill in the document list410. The agreement status interface430displays an expiration date of the document to be Feb. 3, 2022, as identified by the expiration identifier230. The agreement status interface430also displays an estimated time for negotiating a new agreement to be four weeks, as generated by the time prediction module235. In the example ofFIG.4, the interface400also displays a document preview so that a user can view the selected document. The re-negotiation widget440may be a button or other interface configuration for confirming a selection. The user can select the re-negotiation widget440to confirm a request to generate an updated version of the agreement document and to initiate a document collaboration process between the user and a counterparty to the agreement, as may be managed by the collaboration module240. Initiating the document collaboration process may include modifying accounts of the user and the counterparty to enable access to the updated agreement document. The components and functionalities of an interface400may be different in different embodiments.

FIG.5illustrates an example process for identifying upcoming expiring agreement documents and predicting a length of time for re-negotiating the agreement, in accordance with an embodiment. In the embodiment shown inFIG.5, the process is performed by the document management system140. In other embodiments, some or all of the steps of the process may be performed by other components of the system environment or may be performed in a different order than that depicted inFIG.5. Additionally, in other embodiments, the process illustrated inFIG.5can include fewer, additional, or different steps than those described herein.

The document management system140identifies510an expiration date associated with each of a set of agreement documents corresponding to a user. For example, the expiration identifier230may identify an expiration date associated with each of a user's agreement documents stored in the data store205. The time prediction module235of the document management system140applies520a machine learning model, such as the negotiation prediction model325, to a first agreement document of the set of agreement documents. The machine learning model is trained on historical agreement documents associated with a same document type as the first agreement document. The model is configured to output a predicted time of negotiation for the first agreement document. At a time more than the predicted time of negotiation for the first agreement document before the expiration date, the document management system140populates530a notification interface presented to the user, e.g., at a client device120, with a reminder notification identifying an expiration date of the first agreement document and the predicted time of negotiation for the first agreement document. In response to a confirmation from the user via the displayed notification, the document generator220of the document management system140generates540an updated first agreement document and the collaboration module240of the document management system140initiates540a document collaboration process between the user and a counterparty. For example, the document may be sent to the counterparty for review and execution.

Clause Updates Based on Edits to Similar Clauses

The document management system140monitors standard clauses included by a user in documents and, when a clause is updated, flags other documents that include the clause. In this way, a user can be notified of other documents that the user may also want to update to include the language of the updated clause. If the difference between the instance of each clause in a flagged document and the updated clause language exceeds a threshold value, the documents may be presented to the user within an interface that enables the user to automate the process of updating the language of the outdated clause instances in the documents. Referring back toFIG.2, a system for updating similar clauses includes the edit detector245and the document comparison module250.

The edit detector245monitors changes made to documents to detect when clauses have been edited. In various embodiments, the edit detector245may detect changes to edits in different ways. In some cases, the document management system140may receive updated documents from a client device120, such as documents for analysis110. The edit detector245may compare a new version of a document received from the client device120with the same document already stored on the database205to determine if any edits have been made to clauses in the document. In one embodiment, a user of a client device120can make edits to a document by interacting directly with a document editing interface sent to the client device by the document review module225. The edit detector245my detect when a change is made to the language of a clause through the document editing interface. In some embodiments, identifying an edit made by a user to the language of a clause includes detection of an edit made by the user to language to a clause within an agreement document that is being negotiated by the user during a negotiation process. An edited clause may include, but is not limited to, a name change of an entity listed in the clause, a change to a quantity or numerical amount within the clause, a change of jurisdiction or geographic location described in the clause, a change to a liability within the clause, a change to a date within the clause, a change to a proper noun within the clause, or a change to legal terminology within the clause.

The document comparison module250identifies a subset of documents stored in the database205that include an outdated instance of the updated clause. To identify the related documents, the document comparison module250queries the corpus of documents stored in the database205that are associated with the user. From this corpus of documents, the document comparison module250reviews the contents of the documents and identifies the related documents having the original instance of the clause (i.e., as it was before it was edited). The document comparison module250compares the edited clause with the original instance of the clause in the identified related documents and determines whether to include each of the related documents in a subset of documents that will be suggested to a user for editing to include the updated clause (i.e., a subset of the documents that include an outdated instance of the clause). In one embodiment, to determine the subset of documents, the document comparison module250identifies the subset of documents that include instances of the original clause that differ from the edited version of the clause by more than a threshold amount. As one example, a threshold of a difference of at least 10 characters may be required between the edited clause and the version of the original clause in another document for the other document to be included in the subset of related documents. Once the subset of documents is determined by the document comparison module250, the document review module225can modify a document interface presented to the user at a client device120to include an interface element that identifies the edited clause and that also displays the identified subset of document that include the original, now outdated, clause. The user may interact with the document management system140via the interface presented by the document review module225to select one or more of the identified documents for replacement of the outdated clause with an instance of the updated clause, and the selected documents may be updated by the document generator220.

FIG.6illustrates an interface600of a document editing dashboard for reviewing similar clauses across separate documents, in accordance with an example embodiment. In the embodiment shown, the interface600includes a document list610, a first interface area620, a second interface area630, markup indications640, and a confirmation widget650. The example interface600shows a document list610of documents that the document comparison module250identified as a subset of the documents associated with the user that include a clause that matches or nearly matches an edited clause in a document of the user. The example interface also includes a first interface area620that displays the edited clause and a second interface area630that displays a version of the original outdated clause in another document. In various embodiments, the first interface area620and/or the second interface area630includes markup640or another indication (e.g., highlighting) of the differences between the edited clause and the outdated original clause. In the example ofFIG.6, the first interface area620shows text from clause 10.1 of a document called Contract1.docx. The document list610lists documents that the document comparison module250has identified as having the same or a similar original clause (e.g., clause 10.1). In the example, document list610, document AgreementC.docx is selected, as indicated by the dotted fill. Consequently, text from the related original clause in AgreementC.docx is displayed in the second interface area630. The markup640in the first interface area620shows the edits that were made to the clause in Contract1.docx that make it differ from the original clause language which is still present in AgreementC.docx. In the example interface600ofFIG.6, a user can select among the documents in the document list610using checkmarks in check boxes to indicate one or more documents should be modified by the document generator220to include the updated clause language. To confirm the documents for modification, the user can select the confirmation widget650. The components and functionalities of an interface600may be different in different embodiments.

FIG.7illustrates an example process for flagging clauses in documents for updating when a similar clause is updated in another document, in accordance with an embodiment. In the embodiment shown inFIG.7, the process is performed by the document management system140. In other embodiments, some or all of the steps of the process may be performed by other components of the system environment or may be performed in a different order than that depicted inFIG.7. Additionally, in other embodiments, the process illustrated inFIG.7can include fewer, additional, or different steps than those described herein.

When a user edits a document, the edit detector245of the document management system140identifies710an edit made by a user to language of a first clause associated with the user. The document comparison module250of the document management system140queries720a corpus of documents associated with the user to identify a set of documents that include an instance of the first clause. The document comparison module250compares730the edited first clause with the instance of the first clause in each of the set of documents to identify a subset of the set of documents that include an outdated instance of the first clause. Based on the identified subset of documents, the document review module225modifies740a document interface presented to the user to include an interface element that identifies the edited first clause and the subset of documents that include an outdated instance of the first clause. The user may interact with the modified interface to view the differences between the document clauses. In response to a confirmation interaction by the user via the interface element, the document generator220of the document management system140modifies750(e.g., edits) each of one or more of the subset of documents by replacing the outdated instance of the first clause in the document with the edited first clause. For example, each of the subset of documents that the user selected may be edited to include the updated clause language.

Machine-Learned Document Action Suggestions

The document management system140uses machine learning models to identify a type of document with which a user is interacting and to suggest actions that can be taken on the document. Examples of actions that may be taken with respect to a document include replacing text with fields, replacing clauses with pre-approved versions of the clauses, and synchronizing the document with a third-party system. Action options may be based on actions taken by similar users, actions done to similar documents, and actions taken in similar contexts (e.g., at similar times of the year). In one embodiment, the document management system140presents the recommended actions within an interface that enables a user to view and select desired actions from a list of suggested actions and that enables the user to automate the performance of the actions if requested. Referring back toFIG.2, a system for suggesting document actions includes the document type predictor255, the action store260, and the action identifier265.

The document type predictor255applies trained machine learning models to predict a type of document that a user is editing or otherwise interacting with. In one embodiment, the document type predictor255uses a document type prediction model825, which takes in a document, a partial document, and/or a set of document features as input and outputs a predicted type of the document. The document type predictor255may analyze a document to determine a set of associated document features. In some embodiments, information about features associated with a document may additionally or alternately be stored as metadata in relation to the document at the database205. Document features identified by the document management system140may include terms used within the document, clauses used within the document, images within the document, entities associated with the document, permissions associated with the document, actions taken on the document, templates used to generate the document, characteristics of the user, and characteristics of entities associated with a document. Additional information about the training and application of the document type prediction model8125is included in reference toFIG.8below. The document type prediction generated by the document type predictor255may be stored in the database205as metadata associated with the document. In some cases, the document type predictor255may analyze a document upon receipt of the document at the document management system140or the document type predictor255may update document type predictions by reanalyzing documents stored in the database205periodically or as additional training data is received to update the document type prediction model825. In some embodiments, the document type predictor255analyzes a document or partial document to determine the type when the user is actively interacting with the document, for example, via an editing or review interface provided by the document review module225. The document type prediction generated by the document type predictor255is provided to the action identifier for use in determining appropriate actions to suggest to a user with respect to the document.

The action store260stores information about actions that the document management system140can perform on documents. This may include possible actions that can be performed, as included, for example, in the action store by a system administrator or other user. Examples of actions that can be performed for a document include replacing text with fields, adding signature fields, replacing text with pre-approved versions of clauses, synchronizing the document with an external document system, changing a tense used in some or all sections of the document, populating fields with data from external data sources, and providing the document for review or signature to an entity associated with the document. The action store260may additionally store related information and metadata about possible actions. For example, the action store260may store information about which actions are performed on different document types, and how often certain actions are performed on different document types. Other metadata stored by the action store260could include common combinations of actions that have been taken by users on documents, a number of users that have performed certain action types, a number of times an action has been taken on documents, and characteristics of users that take perform certain actions with respect to a document type.

The action identifier265identifies a set of actions that can be taken on a document that user is currently editing or reviewing. The set of actions can then be suggested to the user via an interface presented at a client device120as generated by the document review module225. The action identifier265identifies a set of actions that can be taken on the document by accessing the document type of the current document as predicted by the document type predictor255. The action identifier265then accesses the action store260to obtain a set of actions that have been taken or can be taken on documents of the same type. In some embodiments, actions taken on other documents of the document type include actions taken by the same user on other documents of the document type or actions taken by users with one or more characteristics in common with the user. In some embodiments, actions taken on other documents of the document type include actions taken in the past by at least a threshold number of users or actions that have been implemented at least a threshold number of times on other documents of the document type.

FIG.8is a flow diagram illustrating training and application processes for a document type prediction model825, in accordance with an example embodiment. The document type prediction model825takes, as input, a document820(or a partial document or list of document features associated with the document820), and outputs a predicted type of the document. The document type prediction model825is trained to predict a type of a document using information about sample documents, document features and document types.

The model generator210trains the document type prediction model825using a set of training data800. In some cases, multiple document type prediction models825may be trained and stored in the model store215for use by the document type predictor255. For example, the model store215may store a separate document type prediction model825in association with each individual user or in association with separate entities using the document management system140. The document type prediction model825is trained to identify correlations between document types and document features and the document type of a document may be determined based on an input set of document features using the identified correlations. The training data800may include sample documents805, sample document features810, and sample document types815. In various embodiments, the training data800may include additional or different training inputs not shown inFIG.8.

The training data800may be provided via client device120to the document management system140. In other embodiments, the document management system140may automatically collect the sample documents805, the sample document features810, and the sample document types815to add to the training data800as it processes documents. In other embodiments, a user of a client device120associated with the document management system140(e.g., a system administrator) may manually input or curate a subset of the training data800. It should be noted that the training data800may include historical document data and/or sample documents that are generated and labeled for the purpose of training the document type prediction model825.

Sample documents805used as a subset of the training data800may include the text of documents and metadata associated with the documents. The training set of sample documents805may include documents associated with a particular user, documents associated with an entity or with another user associated with the user, or documents associated with users having one or more characteristics in common with the user. For example, if the model generator210trains a model for each user, then it may use documents associated with similar users for training. In addition to the text and input fields present in a document, the metadata stored in association with a sample document may include document file type, file size, languages within the document, region in which the document originated, characteristics associated with the sending and receiving party of the document (e.g., size, industry, location of headquarters, revenue, corporate structure), types or categories of information or passages within the document, and the like.

The sample document features810used for training the machine learning models may include descriptions of document features or may include annotations of sample documents805as having certain document features. Document features810may include specific terms used within a document, clauses within a document, images within a document, entities associated with a document, permissions associated with the document, actions taken on a document in the past, templates used to generate a document, characteristics of a user associated with the document, characteristics of entities associated with a document, a word count, the presence of signature lines, and specific data collection fields included in the document, among other items.

The sample document types815include annotations to the sample documents805and to the sample document features810that indicate the type of document with which the documents or features can be associated. For example, this may include a document type label associated with a sample document805. In some cases, training information about document types may be input manually by a user of a client device120associated with the document or by a system administrator.

The model generator210uses supervised or unsupervised machine learning to train the document type prediction models825using the training data800. Different machine learning techniques may be used in various embodiments, such as linear support vector machines (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps. The training of a document type prediction model825helps the model to identify correlations between document features and document types. That is, training a document type prediction model825enables the document type prediction model825to identify a predicted type830of a document820. In some embodiments, the model generator210trains multiple document type prediction models825, such as a separate document type prediction model825for each user or for each entity associated with the document management system.

A trained document type prediction model825can be applied by the document type predictor255to a document820. In some embodiments, the document type prediction model825also or alternately accepts metadata associated with the document (e.g., document features) as input. The document type prediction model825generates a predicted type830of the document820. The action identifier265can use a predicted document type830to select actions that are likely to be relevant to the document820from the action store260. The document review module225may display the selected actions for a user of a client device120to view and select so that the action can be taken for the document.

FIG.9illustrates an interface900of a document editing dashboard for viewing and selecting suggested actions related to a document, in accordance with an example embodiment. In the embodiment shown, the interface900includes an action list910, and action confirmation widgets920. The example interface900also displays a view of the document (e.g., Contract1.docx) that the user is currently reviewing or editing. The action list910has been modified by the document review module225to present actions selected by the action identifier265as being available actions to perform on the document. The user can confirm which, if any, of the actions for the document management system140to take with respect to the document by selecting one or more of the action confirmation widgets920. An action confirmation widget920may be a button, a drop-down menu, an input field, or another user input component of the interface900. In response to a user selecting one or more of the actions via the interface900, the document management system140performs the selected actions on the document.

FIG.10illustrates an example process for identifying actions that a user can request for a document, in accordance with an embodiment. In the embodiment shown inFIG.10, the process is performed by the document management system140. In other embodiments, some or all of the steps of the process may be performed by other components of the system environment, or may be performed in a different order than that depicted inFIG.10. Additionally, in other embodiments, the process illustrated inFIG.10can include fewer, additional, or different steps than those described herein.

When a user is interacts with a document, for example to edit or review the document, the document management system140can suggest actions related to the document. The document type predictor255of the document management system140applies1020a machine-learning model to a set of features of the document. The machine learning model may be a document type prediction model825that is trained on a training set of documents. Each of the documents in the training set of documents may include information about training document features and may be tagged or labeled with a training document type. The trained machine learning model is configured to output a document type of the document based on the set of features associated with the given document. Based on the document type output by the machine learning model for the document, the action identifier265of the document management system140accesses an action store260and identifies1030a set of actions that can be taken on the document based on the contents of the document and actions taken on other documents of the same predicted document type. The document review module225of the document management system140modifies1040a document interface presented to the user at a client device120to include an interface element that identifies the set of actions that can be taken on the document. The user may then choose to confirm via the interface that the displayed actions should be taken on the document. In response to a selection of one or more of the set of actions via the interface element by the user, the document management system140performs1050the selected actions on the document.

Additional Configuration Considerations

The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like.

Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the patent rights. It is therefore intended that the scope of the patent rights be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the patent rights, which is set forth in the following claims.