Patent Document ID: 9967211
Application ID: 14726569

Base Claim:
1. A computer-implemented method for automatic assessment of machine generated responses, said method comprising: extracting candidate context-message-response n-tuples, by an extraction component of a computing device, from at least one source of conversational data; forming a set of multi-reference responses selected from the candidate context-message-response n-tuples extracted by the extraction component; calculating an assessment metric for the machine generated response, by at least one processor, based on the set of multi-reference responses; and generating a metric score for the machine generated response based on the assessment metric, by the at least one processor, the metric score indicating a quality of the machine-generated response relative to the set of multi-reference responses.

---

Claim 10:
10. The computer-implemented method of claim 1 , wherein the metric score is a score within a scale from zero to one, and wherein generating the metric score further comprises: calculating an amount of word sequence overlap between the machine-generated response and a reference response in the set of multi-reference responses, wherein an overlap of zero indicates no words in common between the machine-generated response and the reference response, and wherein an overlap of one indicates the machine-generated response is identical to the reference response.