Assertion quality assessment and management system

An assertion assessment system is presented. An assertion object that represents a relationship between an assertion and an information source can be bound to a quality measure as an indication of the quality of the assertion object. The assertion assessment system enables different recipients of assertion objects that are linked to information sources to assess assertion quality of the assertion objects. The assertion assessment system manages and tracks these assessments of the assertion objects and provides useful indications to the recipients. Thus, workers who come upon an assertion can make a judgment on the quality of the assertion based on assessment made by other workers, without going through the information source that supports the assertion.

FIELD OF THE INVENTION

The field of the invention is assessing the quality of assertions, and more particularly a system and method for managing and tracking assertion quality.

BACKGROUND

As countries around the world transition from industrial-based economies to knowledge-based economies, it is increasingly important that we develop systems and methods for assessing the quality of knowledge-based processes, workers and their output.

In executing knowledge-based processes, knowledge workers (e.g., employees of an organization, students, teachers, professors, researchers, etc.) frequently read articles of content and make notes, conclusions, impressions, and/or perceptions (collectively as “assertions”) about what they have learned out of what they have read. Examples of assertions include capturing facts or findings, voicing opinions, asking questions and drawing conclusions. In some cases, knowledge workers will rely on these assertions for making decisions, making recommendations to decision-makers or integrating their assertions into new content they create and some cases publish/distribute.

Historically, we have relied on citation-based reference systems to assess the quality of assertions found in written content that are derived-from, or otherwise supported-by, other articles of content. Standards have been developed in various professional domains which promote consistency of creating footnotes, endnotes and bibliographies, facilitating the ability of the reader to verify assertions found in written content.

Citation-based reference systems present several limitations. They require significant time and effort by knowledge workers who create and rely on them, while introducing numerous opportunities for errors and misuses. Examples of processes citation authors have to execute include source identification, citation capture, page referencing, and formatting. Errors in any of these processes can result in the failure of the quality control system, and numerous opportunities for errors are present. These processes usually take place across multiple systems, requiring the transposing of information from one system to another system. In addition, they frequently involve the copying/pasting of information, and may involve paper-based processes which occur offline and are not electronically auditable. The citation reader has to execute several processes as well. They have to read the footnotes or endnotes, use that information to search for the source documents, acquire the source documents, navigate to the appropriate location within the source documents and make an assessment about the veracity of the source document and its author as well as the merits of the fit between the specific section in the source work and the proposition in the authored work.

Citation systems do not provide means for even capturing assessments of assertion quality such that they could be used in a system for assessing the quality and performance of knowledge-based processes, workers and their output. No means to capture assessments, manage them or use them to advance or improve knowledge work.

Efforts have been made to improve on the citation process with the help of computer technologies and the Internet. For example, hyperlinking keywords or key phrases in online articles within Wikipedia® can alleviate some of the manual work associated with citation-based references.

In another example, the “Explore Sources” feature created by DocumentCloud, an open source project sponsored by the Knight Rider foundation, provides readers of a written work direct access to the source documents and excerpts within those source documents, as disclosed in the article entitled “Explore Sources: A New Feature to ‘Show Our Work’” by Al Shaw, published on Dec. 15, 2011. While DocumentCloud reduces some of the manual work associated with creating and linking to references, it does not provide an assertion-quality assessment system—or even a means to capture an assessment.

With respect to rating, U.S. patent publication 2012/0297182 to Hawkins titled “Cipher and Annotation Technologies for Digital Content Devices”, filed Mar. 17, 2012, discloses a mechanism that allows readers to rate annotations (i.e., comments) of a document. However, the annotations described in Hawkins refer to comments made on a document, rather than independent assertion objects that are supported by or referenced to one or more documents. In addition, the rating system disclosed in Hawkins merely allows readers to indicate how much they liked the comments on the annotations and do not provide any meaningful indication of the quality of assertions made by the annotations in relation to its associated literals.

As such, knowledge enterprises and their workers are in need of a better system for assessing the quality of written works.

SUMMARY OF THE INVENTION

The inventive subject matter provides apparatus, systems, and methods in which an assertion object that represents a relationship between an assertion and an information source can be bound to a quality measure as an indication of the quality of the assertion object. According to some embodiments of the invention, the apparatus, systems, and methods enable recipients of assertion objects that are linked to information sources to assess assertion quality, as well as manage and track assertions. Thus, workers who come upon an assertion can make a judgment on the quality of the assertion without going through the information source that supports the assertion. In one aspect of the invention, an assertion assessment system is presented that provides the mechanisms to obtain quality measures and bind the quality measures to different assertion objects.

DETAILED DESCRIPTION

FIG. 1illustrates an assertion assessment system100of some embodiments. As shown, the assertion assessment system100comprises an assertion database110for storing assertion objects and an assessment tracking engine105. In some embodiments, the assessment tracking engine105is communicatively coupled with the assertion database. The database110of some embodiments is implemented in a non-transitory permanent data storage such as a hard drive, a flash memory, etc. In some embodiments, the database110can be a file system, database management system, a binary large object (BLOB), a document, a table, etc. In some embodiments, the database110stores assertion objects that are associated with a plurality of different assertions made by users.

As shown in the figure, the assertion database110stores assertion object135and assertion object140. Each of the assertion objects135and140represents a relationship between an assertion and an information source. For example, some assertion objects can represent a fact or a point that is supported an information source. Some assertion objects can represent opinion or a conclusion that is derived from an information source. Other assertion objects can represent an observation or a perception that is based on an information source. In some embodiments, the assertion objects can be implemented as metadata object having similar structure and relationship among other metadata objects as described in co-owned U.S. patent application 61/739,367 entitled “Metadata Management System”, filed Dec. 19, 2012.

Each assertion object also includes a set of attributes.FIG. 2illustrates an example assertion object in more detail. Specifically,FIG. 2shows assertion object135and assertion object140that are stored in the assertion database110.FIG. 2also illustrates a set of attributes that is stored within the assertion object135. As shown, assertion object135includes an assertion ID205, an assertion type210, assertion content215, an author identifier220, a creation date225, a last modified date230, a source type235, a source identifier240, one or more assertion quality measures245, and right policy data250. These attributes only represent examples of the kinds of attributes that can be included within an assertion object. The assertion objects of some embodiments can add attributes to or remove attributes from this set to better suit a particular situation.

The assertion ID205is used to uniquely identify an assertion object. It can be used as a reference identifier when it is referenced by another assertion object. It can also be used for storing the assertion object and retrieving the assertion object from the database110.

The assertion type210of an assertion object can be used to indicate a type of the assertion. As mentioned above, each assertion object represents a relationship between an assertion and an information source (e.g., an opinion, conclusion, perspective, etc.). Thus, the assertion type210of some embodiments can indicate an assertion type of the assertion object.

The assertion content215stores the “assertion” of the assertion object. In some embodiments, the content is a word, a phrase, a sentence, a paragraph, or an essay. The assertion (or the assertion content) is generated by a user who has read another piece of content (i.e., the information source). The user then creates the assertion (e.g., an opinion, a conclusion, an observation, a point, an asserted fact, etc.) based on the information source. In some embodiments, the information source can be at least a portion of an article, a publication, a book, a website, or even another assertion.

The author identifier220identifies the author of the assertion. The identifier can be a name, a number (e.g., social security number), or a string of characters. The assertion assessment system100of some embodiments can include another database that stores information of different authors. The system100can then retrieve the author's information by querying the database using the author identifier.

The creation date225and the last modified date230indicate the date that the user created the assertion object and the date that the user last modified the object, respectively.

The source type235indicates the type of source information that is associated with this assertion object. For example, as mentioned above, the information source can be a book, an article, a website or even another assertion. The source type235can contain information that indicates the type of the source information.

The source identifier240identifies the information source that is associated with the assertion object. As mentioned above, the information source can be another assertion, which assertion object is also stored in the assertion database110. In this case, the source identifier240can be the assertion ID of another assertion object. In other cases, the source identifier240can be an identifier of a document ID such as a digital object identifier (DOI). The source identifier240can also be a pointer that directly points to another object within the assertion database110.

In some embodiments, the assertion object can include more than one information sources (e.g., when an assertion is derived from a combination of more than one information sources). In these embodiments, the assertion object can store more than one source type/source identifier pairs.

The assertion quality measure(s)245stores one or more assertion quality measures that are bound to the assertion object. As mentioned above, the assertion quality measure indicates the quality of the assertion or the quality of the relationship between the assertion and its information sources. In some embodiments, the quality measure can be quantified as a score on a pre-defined scale (e.g., numeric scores, categorical scores, etc.). In some instances, there is more than one aspect of an assertion in determining the quality of the assertion/the relationship, such as how well the assertion is supported by the information source, how accurate the assertion sound in view of common sense, etc. Thus, the assertion object can store more than one assertion quality measures for quantifying these different aspects of the assertion.

Right policy data250includes information that indicates which users have access to the assertion object. In some embodiments, it can include a list of users who have access to the assertion object (i.e., a white list), or a list of users who are excluded from accessing the assertion object (i.e., a black list). In other embodiments, it can indicate a specific access level (e.g., top security, public, group, etc.) so that only users who have clearance of a specific access level can access the assertion object.

When an author creates an assertion object, some or all of these attributes are generated automatically based on information collected or given by the author. At least some of these attributes can be updated or modified during the lifetime of the object. In addition, each assertion object is distinctly manageable apart from its information source. For example, the assertion object135can be retrieved from the assertion database independent of its information source. The user can view and modify the content of the assertion object independent of the information source (e.g., viewing the quality measure, updating the quality measure, etc.). The assertion object can also be independently published (either in paper form or digital form) without referring to the information source.

FIG. 3illustrates a graph300that represents the relationships among an example set of assertion objects and source document objects. As shown, the graph300includes source document objects305and310. Source document objects305and310can be any type of literals, such as books, publications, articles, websites, etc. The graph300also includes assertion objects315-360. As mentioned before, an assertion can be created based on a literal (e.g., a book, a publication, etc.). In this example, the graph300shows that assertion objects315,320, and325all point to the literal represented by source document object305. Similarly, assertion objects330and335both point to source document object310.

In addition to, or instead of creating an assertion based on a literal, an assertion can also be created based on other assertion(s). As shown in the graph300, assertion object340identifies assertion objects315and320as its information source, indicating that the assertion object340is generated/derived based on assertion objects315and320. Similarly, assertion objects345points to assertion object320and325as its information source, and assertion object350points to assertion object330and335as its information source.

Furthermore, an assertion can also be associated with (directly or indirectly) more than one literal. For example, assertion object360points to assertion objects345and350as its information source. In this case, assertion objects345and350are indirectly associated with different literals—source document object305and source document object310, respectively. In some embodiments, a graph such as graph300inFIG. 3represents a synthesis structure of a knowledge that is derived from one or more information sources.

Referring back toFIG. 1, the assessment tracking engine105includes an assertion management module115, a quality measure module120, an assertion a user interface module125, and an output interface130. The user interface module125communicates with computing devices145,150, and155over a network (e.g., a local area network, the Internet, etc.). Users behind the computing devices145,150, and155interact with different assertion objects through the user interface module125of the assessment tracking engine105. For example, one of the functions a user can perform on an assertion object is to rate the quality of an assertion being viewed.

FIG. 4illustrates an example user interface400that presents a user with information contained within an assertion object. In some embodiments, the user interface400can be generated by the assessment tracking engine105and presents at the user computing devices145,150, and155through the user interface module125. As shown inFIG. 4, the interface400displays different attributes of an assertion object. For example, the interface400displays assertion content405which recites “Hurricane Sandy, nicknamed ‘Frankenstorm,’ was the strongest, largest storm to hit New York City in recorded history. The interface400also presents the assertion type410of the assertion object as a “fact”.

In addition, the interface400displays information about the author of the assertion415. In this example, it displays the name, title, qualification/credentials, and some information about the quality/reliability about this author. In this instance, the quality of the author is represented by a point system, and this author415is shown to have 235 points. The quality of the author can be affected by the rating of all assertions that he/she publishes. For example, the author will have a high rating when his/her assertions in general receive high quality ratings. Conversely, if the author's assertions in general receive low quality rating, he/she will have a low quality rating.

The interface400also allows a user to insert comments and presents comments that have already been inserted by other users. Furthermore, the interface400allows a user to rate the assertion. In this example, the interface400allows a user to rate an assertion based on a star system, on a scale of one to five stars. As shown, the interface400includes a rating interface410that allows a user to select anywhere between one to five stars for rating the quality of the assertion. One should appreciate that an assertion rating can include a multi-valued rating, where each value can correspond to different aspects of the assertion. For example, the assertion could be rated based on veracity, quality of citations, quality of the proposed argument or logic, or other factors.

As mentioned above, each assertion object includes one or more information source on which the assertion is based. In this example, the interface400shows that the assertion is based on three information sources represented by icons415,420, and425. The information sources can be another assertion or a literal. In some embodiments, the user can retrieve and view these information sources by selecting the icons415,420, or425. If the information source is an assertion object, the interface400can open up another assertion interface that is similar to interface400and presents information of that assertion object. If the information source is a literal, the interface400of some embodiments can open up a different kind of interface to present information of the literal. In some cases, the information source includes only a portion of a complete document, article, or website. In these cases, the new interface can present only the portion of the literal that is being cited/used by the author. Alternatively, the new interface can present the entire literal and highlight the portion of the literal that is being cited/used by the author. This way, the user can trace the chain of sources that is being used for the author to arrive at the assertion.

For example, referring back toFIG. 3, if the user is viewing an assertion that corresponds to assertion object360, the interface400will present the information sources as assertion objects345and350. The interface400will allow the user to select any of the source assertion objects by selecting the information source icons. If the user selects to view the source assertion object345, another interface similar to the interface400will be presented to the user. The user will also be able to see that assertion object345refers to (or cited) information sources that correspond to assertion objects320and325, through the user interface. The user can continually trace the information source chain of the assertion object360by traversing the assertion graph structure (such as graph300). According to the graph300, the information source chain from the perspective of assertion object360includes assertion objects360,345,350,320,325,330,335, and literal objects305and310.

After the user has verified the sources of the assertion through tracing the information source chain, the interface400allows the user to rate the quality of the assertion. As shown inFIG. 4, the interface400provides a rating interface410for a user to rate the quality of the assertion being presented in view of its information source(s). In this example, the rating interface410is presented as a five-star scale in which the user can rate the assertion between one to five stars. Other rating interfaces are also contemplated, such as a sliding bar, a thumb-up/thumb-down interface, etc.

As shown inFIG. 3, an assertion object can have an information source chain that spans across many levels. For example, as illustrated above, assertion object360first points to assertion objects345and350(level one). Assertion objects345and350in turn points to assertion objects320,325,330, and335(level two). Lastly, assertion objects320,325,330, and335point to literal objects305and310(level three). When assessing the quality of the relationship between the assertion and its information sources, one can assess information available to all assertion objects and/or literal objects along the information source chain. Alternatively, one can limit the assessment to the assertion's relationship with only a subset of the levels (e.g., level one only, levels one and two only) of the information source. In these embodiments, the user can only use information that is available to him/her from assertion objects/literal objects from the subset of the levels of the assertion object being assessed.

The quality of an assertion can include one or more dimensions. For example, in one dimension, the user can rate the relevance to the assertions or literals that are being cited. In another dimension, the user can rate the extent to which the underlying information sources support the assertion. In yet another dimension, the user can rate the authors of the underlying information sources. In some embodiments, the interface400can provide a rating interface that allows the user to rate the assertion in different dimensions.

Referring back toFIG. 1, when one or more users interact with the assertion object (e.g., rating the assertion, tracing the information source chain, providing comments, etc.), the data generated from the interaction (the interaction data) is sent to the assessment tracking engine105via the user interface module125. After receiving the interaction data, the user interface module125forwards the interaction data to the assertion management module115.

Based on the interaction data, the assertion management module115determines the assertion object that is associated with the interaction, and retrieves the assertion object from the assertion database110. The assertion management module then sends the interaction data that is relevant for determining the quality measure to the quality measure module120for determining a quality measure for the assertion object.

In some embodiments, the assessment tracking engine105is configured to determine the quality measure of an assertion object based on one or more factors. For example, the assessment tracking engine105can determine the quality measure based solely on the user's rating input or any other interaction data. It is contemplated that the assertion object can be viewed and rated by the author of the assertion object and many different users who have access to view the assertion object. Interaction data of the same assertion object from different users can be received by the assessment tracking engine105over time. When new interaction data is received at the assessment tracking engine105, a quality measure may have already been created and bound to the assertion object.

If there is a quality measure bound to the assertion object already, the quality measure module120can use the new interaction data to update the quality measure. Alternatively, the quality measure module120can retrieve all interaction data that has been previously received at the assessment tracking engine105(they can be stored in a database coupled to the engine), and determine the quality measure based on the existing interaction data and the new interaction data.

This way, the quality measure module120can also apply different weights to interaction data generated at different time. As an example, the quality measure module120can be configured to give more weight to recent interaction data than old interaction data. In addition, the quality measure module120can also track the users who have interacted with the assertion object and give different weights to users with different characteristics (e.g., demographics, cultural background, citizenship, residency, etc.)

When the assessment tracking engine105stores all interaction data that have been received for an assertion object at different times, the assessment tracking engine105can also be configured to analyze this data to generate trend information (e.g., do users being to like/approve the assertion more as time goes by) and other metrics related to the temporal aspect of the assertion rating. The trend information and other temporal metrics can also be used by the quality measure module120to generate/update the quality measure(s) of the assertion object.

In some embodiments, the assessment tracking engine105can also retrieve information from the assertion object and information from the assertion object's information source chain to determine the quality measure. For example, the assertion management module115can retrieve not only the assertion object with which the user is interacting, but also assertion objects and literal objects that are part of the information source chain of the assertion object. The assertion management module115can then send relevant information (e.g., quality measures of the assertion objects and literal objects, author ID and ratings of the assertion objects and literal objects, number of assertion objects and literal objects on the information source chain, creation/modification date of the assertion objects and literal objects, etc.) to the quality measure module120. The quality measure module120can then analyze the relevant information to determine the quality measure for the assertion object.

In some embodiments, the quality measure module120is configured to determine whether a loop exist within the information source chain of the assertion object. A loop exists when the traversal of the information source chain leads back to an assertion object/literal object that has already been examined. For example, if assertion object325points back to assertion object345instead of literal object305, a loop exists. The quality measure module120can be configured to update the quality measure (i.e., negatively affects the validity of the assertion) based on existence of loops in the information source chain.

Similar to user generated asserting ratings, the assertion management module115can selectively retrieve only a subset of the assertion objects/literal objects within the information source chain (e.g., select objects from one particular level or a subset of levels), and use only that information to determine a quality measure for the assertion object.

As mentioned above, the quality of an assertion can be measured in multiple dimensions. As such, the assessment tracking engine105of some embodiments is configured to generate multiple quality measures for an assertion object to represent the different dimensional quality of the associated assertion. For example, the assessment tracking engine105can generate an appropriateness quality measure to represent how appropriate or relevant the assertion is in view of the cited information sources. As another example, the assessment tracking engine105can generate a validity quality measure to represent how well the cited information sources support the validity of the assertion.

Once the quality measure module120generates the one or more quality measures for the assertion object, the quality measure module120sends the quality measures to the assertion management module115. The assertion management module then binds the generated quality measures to the assertion object by saving/updating the quality measure attributes of the assertion object.

In some embodiments, the assertion management module115also configures an output device (such as output device160) via the output interface130to present the quality measures of the assertion object. As shown inFIG. 4, the interface400generated by the assessment tracking engine105includes an area430to present the quality measure(s) of the assertion object. In this example, the assertion object has a quality score of four stars.

The assessment tracking engine also comprises a quality input interface for receiving data associated with user interactions with the assertion objects over a network