Patent ID: 11972442
Assignee: WEVO, INC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 9:
10. One or more non-transitory computer-readable media storing instructions which, when executed by one or more hardware processors cause:
monitoring, by a network service using at least one trained machine learning model, activity of respondents of a user experience test or survey for fraudulent behavior, wherein the at least one trained machine learning model has been enabled to automatically remove respondents from a set of accepted respondents for the user experience test or survey;
generating, by the network service using the at least one trained machine learning model based at least in part on one or more inputs of a respondent, a score representing an estimated likelihood that the respondent is engaging in fraudulent behavior with respect to the user experience test or survey;
responsive to determining that the score exceeds a threshold, removing the respondent from the set of accepted respondents for the user experience test or survey;
detecting, by the network service responsive to removing the respondent, that a rate at which the at least one trained machine learning model has detected fraud has exceeded a threshold; and
responsive to detecting that the rate at which the at least one trained machine learning model has detected fraud has exceeded the threshold, pausing or disabling, by the network service, use of the at least one trained machine learning model for automatic respondent removal, wherein pausing or disabling use of the at least one trained machine learning model for automatic respondent removal includes pausing or disabling additional model training and application of the at least one trained machine learning model to automatically remove respondents of the user experience test or survey.