SURVEY PANELIST UTILIZATION

A facility for assembling a list of survey qualification questions to present to a person is described. The facility first initializes the list to be empty. The facility adds to the initialized list a survey qualification question for a survey elected by the person. Until the list reaches a predetermined size, repeats the following: for each of a group of candidate survey qualification questions, for each of the survey qualification questions on the list, the facility determines an asymmetric overlap score for the candidate survey qualification question with the survey qualification question on the list; among the determined asymmetric overlap scores, the facility determining a lowest one, and moves one or more survey qualification questions from the group of candidate survey qualification questions to the list.

BACKGROUND

Market research surveys collect feedback from the target audience for a product or service to understand their characteristics, expectations, and requirements.

It is common for companies that have a product or service in the market or are considering launching one to (1) prepare a sequence of questions soliciting audience feedback that constitutes the survey; (2) specify conditions a respondent must satisfy in order to be able to take the survey and have their responses to its questions included in the survey result; and (3) engage one or more market survey platforms to administer the survey to respondents who satisfy the conditions, and report the results.

Market research platforms typically accept surveys from multiple clients. The platforms assemble pools of people (a panel) who are interested in taking market research surveys—often in exchange for compensation in various forms—and provide a web interface and/or mobile app interface in which all members can select and take surveys relating to different products and services from different clients of the platform.

In particular, it is common for these platform interfaces to display a number of in-process surveys to a panel member; receive the panel member's selection of one of the displayed surveys; pose one or more qualification questions as a basis for determining whether the panel member satisfies the conditions specified for the selected survey; if the panel member satisfies the selected survey's conditions, present the survey to the panel member for completion; and include the panel member's responses to the selected survey in the survey result.

DETAILED DESCRIPTION

The inventors have identified significant disadvantages of conventional approaches to administering market research surveys (“surveys”) in market research platforms (“platforms”). Specifically, the inventors have recognized that these conventional approaches make inefficient use of panel members' time, and are often frustrating to panel members, in some cases causing them to resign or otherwise disengage from the panel. This is because, under the conventional approach, a panel member may have to repeat the following set of steps a number of times before they are able to take a survey: (1) review a list of available surveys; (2) select one of the listed surveys; (3) answer qualification questions for the selected survey; and (4) learn that they did not satisfy the survey's conditions, and must select another survey.

In response to recognizing these disadvantages, the inventors have conceived and reduced to practice a software and/or hardware facility for improved survey administration (“the facility”). When a panel member selects a survey from a list of available surveys, the facility constructs a set of qualification questions to present to the user. Each qualification question in the set corresponds to a different survey and serves as a basis for determining whether the panel member is qualified to take the survey. One of the questions corresponds to the survey selected by the user. The facility selects the other questions based on factors that include (1) disfavoring questions correctly answered at a high rate by panel members that have correctly answered questions already included, especially where a large number of users have answered both, and (2) favoring earlier target dates for finishing the surveys or other business considerations among surveys. Selecting qualification questions for the set that have low correlation in accordance with the first of these two factors is referred to herein as “asymmetric overlap analysis.” If the panel member answers the question for the selected survey correctly, the facility administers the selected survey to them. If they do not, but answer questions for one or more other surveys correctly, the facility administers one of these other surveys to them.

In some embodiments, the facility maintains a list of surveys that it presents to each panel member for selection, which includes only a proper subset of all surveys available on the platform. In some embodiments, this list is of fixed size, such as 8 or 12 surveys. In some embodiments, in addition to removing from the list surveys taken by the panel member as they are completed, the facility also removes from the list surveys whose qualification questions the panel member has answered incorrectly. Each time a survey is removed from the list, the facility replaces it with another survey available on the platform that the panel member has not completed, nor answered the qualification question incorrectly.

By performing in some or all of the ways described above, the facility efficiently assigns panel members to the surveys they will qualified for and have a greater chance to complete. This assists with panel member retention and engagement because the panel member will be able to redeem rewards for completed surveys. The facility also provides a large measure of randomness in the assignment of surveys to panel members, providing fairness among different companies placing surveys on a platform. The facility also often improves the level at which business considerations of the platform's operator are satisfied.

Also, the facility improves the functioning of computer or other hardware, such as by reducing the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be permitted by less capable, capacious, and/or expensive hardware devices, and/or be performed with lesser latency, and/or preserving more of the conserved resources for use in performing other tasks. For example, by reducing the average number of interactions a user must make with a client computing system in order to begin taking each survey, the facility reduces the number of processing cycles consumed by this process on the client computing system, and, in some cases, on a server computing system supporting the operation of the client computing system.

FIG. 1is a network diagram showing a sample environment in which the facility operates in some embodiments. A number of client devices110,120, and130are connected via the Internet140or another network to one or more servers150that operate a survey platform. Some of the client devices—such as client device110—execute a browser that interacts with the survey platform software on the server on behalf of a panel member using the client device, while other client devices—such as client device130—execute a specialized mobile app or desktop application that interacts with the survey platforms software on the server on behalf of a panel member using the client device.

FIG. 2is a block diagram showing some of the components typically incorporated in at least some of the computer systems and other devices on which the facility operates, including the devices shown inFIG. 1. In various embodiments, these computer systems and other devices100can include server computer systems, cloud computing platforms or virtual machines in other configurations, desktop computer systems, laptop computer systems, netbooks, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, etc. In various embodiments, the computer systems and devices include zero or more of each of the following: a processor101for executing computer programs and/or training or applying machine learning models, such as a CPU, GPU, TPU, NNP, FPGA, or ASIC; a computer memory102for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel, and device drivers; a persistent storage device103, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive104, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer-readable medium; and a network connection105for connecting the computer system to other computer systems to send and/or receive data, such as via the Internet or another network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like. While computer systems configured as described above are typically used to support the operation of the facility, those skilled in the art will appreciate that the facility may be implemented using devices of various types and configurations, and having various components.

FIG. 3is a flow diagram showing a process performed by the facility in some embodiments to administer surveys to a panel member. In act301, the facility chooses available surveys from which the panel member has not been excluded to display to the panel member.

FIGS. 4A and 4Bare table diagrams showing sample contents of a survey table used by the facility in some embodiments used to store information about active surveys placed on the survey platform. The survey table400is made up of rows, such as rows411-423, each corresponding to a different survey that is active in the survey platform. Each row is divided into the following columns: a survey id column401uniquely identifying the survey to which the row corresponds; a description column402containing text describing the survey; a reward level column403indicating an amount of money, credit, or other reward that panel members will receive for taking the survey; a volume target column404indicating the number of panel members that the company placing the survey on the platform is seeking to have take the survey; a volume completed column405indicating the number of panel members who have taken the survey; a target date column406identifying the date by which the survey is to have been administered to the volume target number of panel members; a fee column407indicating the price that was paid in connection with the placement of the survey; a qualification question column408containing a question that is to be presented to panel members to determine whether they are qualified to take the survey; and a correct answer column409indicating the correct answer to the qualification question. For example, row411indicates that a “home cinema” survey with survey id 58133448 has a reward level of 20 credits, has been taken by 322 panelists out of a target of 750, is scheduled to be completed by Mar. 19, 2021, is the subject of an $8,000.00 fee, and has a qualification question of “Are you in the market for a home cinema system?” with a correct answer of “Yes.”

WhileFIGS. 4A-4Band each of the table diagrams discussed below show a table whose contents and organization are designed to make them more comprehensible by a human reader, those skilled in the art will appreciate that actual data structures used by the facility to store this information may differ from the table shown, in that they, for example, may be organized in a different manner; may contain more or less information than shown; may be compressed, encrypted, and/or indexed; may contain a much larger number of rows than shown, etc.

In some embodiments, the facility performs act301by randomly choosing a number of the surveys in the survey table that have not been excluded for the panel member.FIG. 5is a table diagram showing sample contents of a user survey exclusion table used by the facility in some embodiments to identify surveys from which particular panel members are excluded. The user survey exclusion table500is made up of rows, such as rows511-513, each corresponding to a particular panel member and one survey from which they are excluded. Each row is divided into the following columns: a user id column501identifying the panel member by their user id, a survey id column502identifying a survey from which the panel member is excluded; and an exclusion reason column503indicating a reason for which the panel member is excluded from the survey. For example, row511indicates that the panel member having user id 445564 is disqualified from the survey having survey id 19245684 because this panel member failed the qualification question specified for the survey. Row512indicates that the panel member having user id 445566—who is the primary subject of this example as it continues—has been excluded from the survey having survey id 63474030 because their demographic information does not satisfy a demographic condition specified for the survey. Such demographic conditions can be specified with regard to any demographic attributes of the panel members, such as age, sex, geographic location, occupation, income level, political or religious affiliation, etc. Also, row513indicates that the panel member having user id 445578 is excluded from the survey having survey id 75135998 because this panel member has already completed this survey.

Returning toFIGS. 4A and 4B, in performing act301on behalf of the audience member user id 445566, the facility randomly selects eight of rows411-423in survey table400, excluding the survey in row415, which is the survey from which this panel member is excluded by row512in user survey exclusion table500. The facility randomly selects rows411,414,416,417, and419-422.

Returning toFIG. 3, in act302, the facility displays the surveys chosen in act301to the panel member.FIG. 6is a display diagram showing a sample display presented by the facility in some embodiments to permit a panel member to select an available, unexcluded survey. The display600includes a message610directing the panel member to select a survey. It further contains a number of tiles620each corresponding to a different survey. In particular, each tile contains a name or description of the survey, as well as reward level at which the panel member will be compensated if they complete the survey. For example, tile621represents a “Subscription music services” survey, which has a survey id of 68713452, and a reward level of 15 credits corresponding to row417of the survey table400. In some embodiments (not shown), the survey tiles contain only information that does not give away the subject of the survey, in order to avoid bias that might result from panel members selecting the subject matter of their surveys.

Returning toFIG. 3, in act303, the facility receives the panel member's selection of a displayed survey. In the example, the panel member selects tile621for the subscription music services survey.

In act304, the facility constructs a set of qualification questions to pose to the panel member in response to the panel member's selection of this survey. Details of the facility's performance of act304are discussed below in connection withFIG. 7.

FIG. 7is a flow diagram showing a process performed by the facility in some embodiments to construct a set of qualifications based upon a particular panel member's selection of a particular survey. In act701, the facility creates a list of qualification questions containing only the qualification question for the survey selected by the panel member. In the example, in act701, the facility creates a list of qualification questions containing only the qualification question for the “Subscription music services” survey selected by the panel member, “How many hours a week do you listen to music?” (as shown in row417ofFIG. 4A). In act702, the facility collects a candidate group of qualification questions from among the available qualification questions in the survey table. In various embodiments, this collection omits surveys for various reasons, including: the survey is the one already selected by the audience member, and its qualification question is already in the list; the survey has demographic conditions not satisfied by the audience member; the audience member has already completed the survey; the audience member has answered the survey's qualification question incorrectly; etc. In some embodiments, where the survey table contains a large number of active surveys, the facility randomly selects a subset of eligible surveys from which to collect the candidate group of qualification questions; selects a subset of eligible surveys with the earliest target dates, or other compelling business considerations such as those discussed below in connection with step703; etc.

In act703, the facility moves one or more qualification questions from the candidate group collected in act702to the list of qualification questions based upon business considerations. For example, in some embodiments, the facility moves one or more qualification questions on the basis that their surveys have the earliest target dates, such as a “tropical vacations” survey shown in row419of survey table400, which has the earliest target date among those shown inFIGS. 4A and 4B. In some embodiments, the facility includes in these business considerations the fee that has been or will be paid by the client for the completion of each survey, either (1) in its total amount, (2) distributed over the size of its volume target, or (3) distributed across the size of the unfulfilled portion of its volume target. In some embodiments, the facility includes among the business considerations the number or percentage of completions remaining for each survey. In some embodiments, the facility considers among the business considerations the reward level of each survey. In some embodiments, the facility considers among the business considerations how well-suited the panelist is to each survey, such as having a particular expertise, body of knowledge, or level of experience relevant to certain surveys. In some embodiments, the facility considers among the business considerations how well-suited each survey is to the panelist, such having a length, subject matter, or reward type or amount favored by the panelist. Those skilled in the art will appreciate that additional business considerations may be used, either alone or in combination.

In acts704-706, the facility repeats act705until the list of qualification questions is full—that is, the list contains a predetermined target number of qualification questions, such as seven. In act705, the facility moves to the list of qualification questions the qualification question in the candidate group having the lowest asymmetric overlap score with the qualification questions already on the list. Additional details about the facility's performance of act705are discussed below in connection withFIGS. 8-9.

FIG. 8is a flow diagram showing a process performed by the facility in some embodiments to identify the qualification question in the candidate group having the lowest asymmetric overlap score with the qualification questions already in the list. In acts801-804, the facility loops through each pair of (1) a qualification question on the list, and (2) a qualification question in the candidate group. In act802, the facility determines an asymmetric overlap score between the current pair of qualification questions in each direction. For a particular such pair of qualification questions Qx and Qy, the facility determines asymmetric overlap scores in each direction as discussed below in connection with Equations (1) and (2).

To determine asymmetric overlap between Qx and Qy in the Qx-to-Qy direction, the facility applies Equation (1) above. In particular, the facility divides the number of panel members answering both Qx and Qy correctly by the number of panel members answering Qx correctly. The two terms of this quotient are discussed below in connection withFIG. 9.

FIG. 9is a Venn diagram showing all possible patterns of interaction of panel members with qualification questions Qx and Qy. Set901is all panel members that gave any answer to Qx. Among these is set902of all panel members who answered question Qx correctly. Similarly, set903is all of the panel members who gave any answer to question Qy, and set904is the panel members who gave the correct answer to question Qy. The intersection among the four sets mentioned above are labeled with reference numbers911-918. In terms of the Venn diagram, the numerator of Equation (1) is region911, the intersection between sets902and904. The denominator is set902, the sum of regions911,912, and914.

The facility applies Equation (2) above to determine the asymmetric overlap between questions Qx and Qy in the Qy-to-Qx direction. This is the number of panel members answering both question Qx and question Qy correctly—the same numerator as in Equation (1)—divided by the number of panel members answering question Qy correctly. In terms of Venn diagram900inFIG. 9, the numerator of Equation (2) is region911, the intersection of sets902and904. The denominator is set904, the sum of regions911,913, and915.

Returning toFIG. 8, in some embodiments, the facility caches some or all of the asymmetric overlap scores, such that, for some or all asymmetric overlap scores, they are retrieved rather than calculated in act802. In various embodiments, the facility calculates the cached asymmetric overlap scores either in earlier iterations of act802, or in a separate, comprehensive asymmetric overlap score calculation process (not shown). In some embodiments, the facility maintains the cached asymmetric overlap scores in a table, such as a qualification question correlation table.

In act803, the facility selects the higher score for the pair, that is, max(AOQx,Qy, AOQy,Qx). In some embodiments, the facility caches the selected score of the pair for use in future iterations of act803(not shown). In act804, if additional pairs remain to be processed, the facility continues in act801, else the facility continues in act805. In act805, the facility determines the lowest asymmetric overlap score among those selected in act803. In act806, the facility chooses for moving from the candidate group to the list one or more qualification questions for which the lowest score determined in act805was selected in act803. After act806, this process concludes.

In some embodiments, where the number of joint observations (in terms of the Venn diagram, the intersection of sets901and903, equal to the sum of regions911,912,913, and918) for question Qx and question Qy is below some threshold, such as 10, then the facility assigns an asymmetric overlap score of zero to this pair of questions, without performing the calculations specified in Equations (1) and (2) above. In some embodiments, if a particular question has been answered less than a threshold number of times, such as thirty (for question Qx, set901, equal to the sum of regions911,912,913,914,916, and918), then the facility assigns an asymmetric overlap score of zero to all pairs of questions containing this question, without performing the calculations specified in Equations (1) and (2) above.

In some embodiments, the facility limits the panel member answers to qualification questions that it considers in calculating asymmetric overlap in one or more dimensions. In some embodiments, the facility limits the answers to questions it considers in calculating asymmetric overlap to those panel members in the same demographic group or other audience segment as the audience member for whom asymmetric overlap is being calculated. In some embodiments, the facility considers answers to questions that have been received recently from audience members, such as within a predetermined period of time of the present, or a certain number of the most recent question responses. In various such embodiments, the facility considers the time at which audience members answered a particular one of the two questions; an average time among the times at which each audience member answered the two questions; an earliest or latest time among these two times; etc.

Those skilled in the art will appreciate that the acts shown inFIG. 8and in each of the flow diagrams discussed herein may be altered in a variety of ways. For example, the order of the acts may be rearranged; some acts may be performed in parallel; shown acts may be omitted, or other acts may be included; a shown act may be divided into subacts, or multiple shown acts may be combined into a single act, etc.

Returning toFIG. 3, in act305, the facility administers and grades the set of qualification questions constructed in act304. In act306, the facility stores a panel member's qualification question results in a qualification question result table.FIGS. 10 and 11illustrate this process.

FIG. 10is a table diagram showing initial sample contents of a qualification question result table used by the facility in some embodiments to store the results of administering particular qualification questions to particular audience members. The qualification question result table1000is made up of rows such as rows1011and1012, each corresponding to the combination of a particular panel member and a qualification question that the panel member has answered. Each row is divided into the following columns: User id column1001containing the user id identifying the panel member; a survey id column1002containing a survey id identifying in the survey whose qualification question was administered; a date/time column1003identifying the date and time at which an answer to the qualification question was received from the panel member; and a correct column1004containing an indication of whether the answer received as correct. For example,1011indicates that a panel member having user id 445564 answered the qualification for the survey having survey id 46513513 on Jan. 15, 2021 at 2:22 pm, and answered it incorrectly.

FIG. 11is a table diagram showing subsequent sample contents of the qualification question result table. The contents of table1100reflect the administration of seven qualification questions to the panel member who is the subject of the example, who has user id 445566. Comparing table1100to table1000, it can be seen that the facility has added rows1113-1119, each corresponding to one of the qualification questions administered to the panel member in the example after selecting the subscription music services survey. From rows1113,1114,1115,1117, and1118, it can be seen that this panel member correctly answered the qualification questions for the surveys having survey ids 68713452, 25879632, 17954623, 94765216, and 44685315, respectively. New rows1116and1119, on the other hand, identify surveys whose qualification questions the panel member answered incorrectly.

Returning toFIG. 3, in act307, the facility excludes for this panel member each of the surveys whose qualification questions the panel member answered incorrectly.

FIG. 12is a table diagram showing subsequent sample contents of the user survey exclusion table, as updated to reflect qualification questions answered incorrectly by the panel member in the example. By comparing table1200to table500discussed above, it can be seen that the facility has added rows1214and1215for the panel member in the example, who has user id 445566. These new rows of the user survey exclusion table correspond to rows1116and1119of the qualification question result table shown inFIG. 11.

Returning toFIG. 3, in act308, the facility selects a survey among those to which the correctly-answered qualification questions correspond. In various embodiments, the facility performs act308by selecting one of the surveys randomly; selecting the one of the surveys whose qualification question was added to the qualification list the earliest; selecting one of the surveys on the basis of business factors such as those discussed above in connection with act703discussed above; etc. In the example, the facility selects for administration the subscription movie services survey that was chosen by the panel member, and accordingly was the first survey added to the list of qualification questions. In act309, the facility administers the selected survey. In some embodiments, this involves displaying a sequence of questions, permitting the panel member to provide an answer or other response to each of the questions; recording these responses; and performing various forms of post-processing on these responses. In act310, the facility excludes for this panel member the survey administered in act309. After act310, the facility continues in act301to choose and display another set of available, unexcluded surveys to the panel member.

FIG. 13is a table diagram showing subsequent sample contents of the user survey exclusion table, updated to exclude the administered survey for the panel member in the example. By comparing table1300to table1200shown inFIG. 12, it can be seen that the facility has added row1315, indicating that this user has completed the subscription music services survey having survey id 68713452.

FIG. 14is a display diagram showing subsequent sample contents of the display presented by the facility to permit a panel member to select an available, unexcluded survey. By comparing display1400to display600shown inFIG. 6and discussed above, it can be seen that the facility has replaced tile621for the subscription music services survey, which the audience member completed, with tile1421for a tennis rackets survey; replaced tile623for the home cinema survey whose qualification question the user answered incorrectly with tile1423for a chocolate truffles survey; and replaced tile626for the reverse mortgages survey, whose qualification question the panel member answered incorrectly, with tile1426for a rental cars survey.