Patent Document ID: 9967211
Application ID: 14726569

Base Claim:
1. A computer-implemented method for automatic assessment of machine generated responses, said method comprising: extracting candidate context-message-response n-tuples, by an extraction component of a computing device, from at least one source of conversational data; forming a set of multi-reference responses selected from the candidate context-message-response n-tuples extracted by the extraction component; calculating an assessment metric for the machine generated response, by at least one processor, based on the set of multi-reference responses; and generating a metric score for the machine generated response based on the assessment metric, by the at least one processor, the metric score indicating a quality of the machine-generated response relative to the set of multi-reference responses.

---

Claim 2:
2. The computer-implemented method of claim 1 , wherein extracting candidate context-message-response n-tuples from at least one source of conversational data and forming a set of multi-reference responses further comprises: extracting candidate context-message-response n tuples from the at least one source of conversational data, wherein individual candidate context-message-response n-tuples comprise a human-generated message, a conversational context, and a reference response corresponding to the human-generated message.