Patent Document ID: 9967211
Application ID: 14726569

Base Claim:
1. A computer-implemented method for automatic assessment of machine generated responses, said method comprising: extracting candidate context-message-response n-tuples, by an extraction component of a computing device, from at least one source of conversational data; forming a set of multi-reference responses selected from the candidate context-message-response n-tuples extracted by the extraction component; calculating an assessment metric for the machine generated response, by at least one processor, based on the set of multi-reference responses; and generating a metric score for the machine generated response based on the assessment metric, by the at least one processor, the metric score indicating a quality of the machine-generated response relative to the set of multi-reference responses.

---

Claim 7:
7. The computer-implemented method of claim 1 , wherein a rating of individual multi-reference responses in the set of multi-reference responses is a human-generated rating, and further comprising: accessing the rating of the individual multi-reference responses in the set of multi-reference responses, wherein the rating indicates a quality of the individual multi-references responses relative to a reference multi-reference response.