COMPUTERIZED SYSTEMS AND METHODS FOR AUTOMATED PERFORMANCE OF GROWING BASELINE ASSESSMENTS

Disclosed are systems and methods for an assessment framework that operates to dynamically generate customized surveys for recipient-respondent pairs. The customized surveys can have curated questions dynamically selected and/or provided from recipients based on, but not limited to, which recipients a respondent should give feedback to, how many questions the respondent should answer, and which questions in particular, the respondent should answer. The disclosed framework can automatically and dynamically customize surveys for individual respondents, and/or sets of respondents, according to the selected questions included therein and/or from which recipients they are sent from (or on their behalf).

BACKGROUND

Surveys serve as important resources for entities (e.g., companies) and their managers to collect information from parties (e.g., users or employees, referred to as respondents). In certain circumstances, surveys can be used to drive productivity and enable better decision making.

Current solutions for engaging survey participation from a plurality of networked sources are deficient in that they focus on probabilistic modelling based on past user behaviors in order to predict how respondents will engage, if at all.

SUMMARY

Presently known systems fall short of establishing end-to-end (E2E) solutions that capitalize on real-time data analytics and respondent data that enables customized assessments to be compiled and deployed thereby triggering improved respondent engagement, and enhances big data collection for purposes of optimizing system resources.

The systems and methods disclosed herein provide an improved distributed, E2E assessment framework. The disclosed assessment framework, as discussed in more detail below, is configured to dynamically generate surveys based on two forms of criteria: question selection and question distribution. These criteria enable the framework to formulate and distribute surveys to respondents that have questions that account for: i) which recipients (which is a user that provides or selects questions for a survey and receives the answers) if any a respondent should give feedback to, ii) how many questions the respondent should answer, and iii) which questions in particular, the respondent should answer.

According to some embodiments, the framework operates by performing a dynamic determination of question selection and question distribution. In some embodiments, as discussed in more detail below, question selection corresponds to a determination of which recipients a respondent gives feedback to in a given round, which also provides a basis for a determination of how many questions a respondent should answer for each recipient in a given round. In some embodiments, question distribution corresponds to a determination of which questions in particular should be included in a survey round. In other words, which questions should a survey include that the respondent will answer for a given recipient (e.g., whether the respondent gets the baseline survey for a recipient).

Thus, according to some embodiments, rounds (or iterations) of surveys can be distributed to a set of users. As discussed in more detail below, the types of questions, quantity of questions, and source of questions (e.g., which recipient is sending a respondent a question) can be dynamically determined, which can drive how surveys are compiled for each respondent. In some embodiments, surveys can be dynamically customized for individual respondents, and/or sets of respondents (e.g., departments within a company).

In accordance with one or more embodiments, the present disclosure provides computerized methods for an assessment framework that dynamically determines and distributes surveys to sets of users that include personalized and quantified questions therein.

In accordance with one or more embodiments, the present disclosure provides a non-transitory computer-readable storage medium for carrying out the above mentioned technical steps of the framework’s functionality. The non-transitory computer-readable storage medium has tangibly stored thereon, or tangibly encoded thereon, computer readable instructions that when executed by a device (e.g., a client device) cause at least one processor to perform a method for an assessment framework that dynamically determines and distributes surveys to sets of users that include personalized and quantified questions therein.

In accordance with one or more embodiments, a system is provided that comprises one or more computing devices configured to provide functionality in accordance with such embodiments. In accordance with one or more embodiments, functionality is embodied in steps of a method performed by at least one computing device. In accordance with one or more embodiments, program code (or program logic) executed by a processor(s) of a computing device to implement functionality in accordance with one or more such embodiments is embodied in, by and/or on a non-transitory computer-readable medium.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

For purposes of this disclosure, a “wireless network” should be understood to couple client devices with a network. A wireless network may employ stand-alone ad-hoc networks, mesh networks, Wireless LAN (WLAN) networks, cellular networks, or the like. A wireless network may further employ a plurality of network access technologies, including Wi-Fi, Long Term Evolution (LTE), WLAN, Wireless Router (WR) mesh, or 2nd, 3rd, 4thor 5thgeneration (2G, 3G, 4G or 5G) cellular technology, mobile edge computing (MEC), Bluetooth, 802.11b/g/n, or the like. Network access technologies may enable wide area coverage for devices, such as client devices with varying degrees of mobility, for example.

In short, a wireless network may include virtually any type of wireless communication mechanism by which signals may be communicated between devices, such as a client device or a computing device, between or within a network, or the like.

A client device (UE) may vary in terms of capabilities or features. Claimed subject matter is intended to cover a wide range of potential variations, such as a web-enabled client device or previously mentioned devices may include a high-resolution screen (HD or 4K for example), one or more physical or virtual keyboards, mass storage, one or more accelerometers, one or more gyroscopes, global positioning system (GPS) or other location-identifying type capability, or a display with a high degree of functionality, such as a touch-sensitive color 2D or 3D display, for example.

With reference toFIG.1, system100is depicted which includes UE402(e.g., a client device, as mentioned above), network102, cloud system104and assessment engine200. UE402can be any type of device, such as, but not limited to, a mobile phone, tablet, laptop, sensor, Internet of Things (IoT) device, autonomous machine, and any other device equipped with a cellular or wireless or wired transceiver. Further discussion of UE402is provided below in reference toFIG.4.

Network102can be any type of network, such as, but not limited to, a wireless network, cellular network, the Internet, and the like (as discussed above). Network102facilitates connectivity of the components of system100, as illustrated inFIG.1.

Cloud system104can be any type of cloud operating platform and/or network based system upon which applications, operations, and/or other forms of network resources can be located. For example, system104can be a service provider and/or network provider from where services and/or applications can be accessed, sourced or executed from. In some embodiments, cloud system104can include a server(s) and/or a database of information which is accessible over network102. In some embodiments, a database (not shown) of cloud system104can store a dataset of data and metadata associated with local and/or network information related to a user(s) of UE402and the UE402, and the services and applications provided by cloud system104and/or assessment engine200.

Assessment engine200, as discussed above and below in more detail, includes components for optimizing how surveys or assessments are compiled and distributed to participating users. According to some embodiments, assessment engine200can be a special purpose machine or processor and could be hosted by a device on network102, within cloud system104and/or on UE402. In some embodiments, engine200can be hosted by a peripheral device connected to UE402.

According to some embodiments, as discussed above, assessment engine200can function as an application provided by cloud system104. In some embodiments, engine200can function as an application installed on UE402. In some embodiments, such application can be a web-based application accessed by UE402over network102from cloud system104(e.g., as indicated by the connection between network102and engine200, and/or the dashed line between UE402and engine200inFIG.1). In some embodiments, engine200can be configured and/or installed as an augmenting script, program or application (e.g., a plug-in or extension) to another application or program provided by cloud system104and/or executing on UE402.

As illustrated inFIG.2, according to some embodiments, assessment engine200includes baseline module202, question selection module204, question distribution module206and survey distribution module208. It should be understood that the engine(s) and modules discussed herein are non-exhaustive, as additional or fewer engines and/or modules (or sub-modules) may be applicable to the embodiments of the systems and methods discussed. More detail of the operations, configurations and functionalities of engine200and each of its modules, and their role within embodiments of the present disclosure will be discussed below in relation toFIG.3.

FIG.3provides Process300which details non-limiting example embodiments of the disclosed assessment framework’s operations of dynamically generating customized surveys for respondents. As discussed herein, customized surveys can have curated questions dynamically selected and/or provided from recipients based on, but not limited to, which recipients a respondent should give feedback to, how many questions the respondent should answer, and which questions in particular, the respondent should answer. Thus, as discussed below, Process300provides example embodiments of surveys that are dynamically customized for individual respondents, and/or sets of respondents (e.g., departments within a company), according to the selected questions included therein and/or from which recipients they are sent from (or on their behalf).

According to some embodiments, Steps302-304of Process300can be performed by baseline module202of assessment engine200; Steps306-310can be performed by question selection module202; Step312can be performed by question distribution module206; and Step314can be performed by survey distribution module208.

Process300begins with Step302where a set of recipient-respondent pairs are identified. According to some embodiments, a recipient-respondent pair involves a recipient, which is a user that provides or selects questions for a survey and receives the answers (e.g., a completed survey), and a respondent, which as discussed above, is the answering user to the posited questions in the survey. For example, a recipient-respondent pair can include a Human Resource (HR) manager and an employee at a company, respectively.

According to some embodiments, the set of recipient-respondent pairs can involve a recipient, and a set of respondents selected from a predefined group. For example, a recipient can be paired with employees within a specific department of a company.

According to some embodiments, each respondent within the identified set of recipient-respondent pairs can have a question budget set, which limits the total number of questions each respondent can be asked by the total of recipients. For example, if a respondent is paired with 2 recipients, and the budget is X questions for the respondent, then each recipient may only be able to ask the recipient their share of X questions (e.g., X/2).

According to some embodiments, a question budget can vary dependent on a feedback cadence (e.g., how often a respondent is asked to respond to a survey). In some embodiments, a feedback cadence can correspond to a predetermined time period, for example: weekly, bi-weekly, monthly, quarterly, yearly, and the like.

Thus, in some embodiments, dependent on the feedback cadence and the question budget, the number of questions within a survey round (or per iteration) can be further limited. For example, if the question budget for a respondent is 48 questions per year, and the feedback cadence is quarterly, the respondent may only be asked 12 questions per time they are issued a survey to respond to.

In Step304, a baseline assessment for each recipient-respondent pair is performed. According to some embodiments, a baseline assessment includes a set of questions put forth on behalf of the recipient for which the respondent has a predetermined time to answer (e.g., 2 weeks). In some embodiments, the survey of the baseline assessment can comprise a criteria that requires all of its questions be answered. In some embodiments, the questions included in the baseline assessment can be randomly selected (e.g., by a randomization algorithm executing in conjunction with engine200); and in some embodiments, the recipient can select at least a portion or all of the questions.

In some embodiments, the baseline assessment can serve as an initial survey between a recipient-respondent pair. In some embodiments, if a respondent-recipient pair has already been subject to a baseline assessment, then engine200may retrieve the information from the previously issued baseline assessment rather than reiterate a survey between the established pair. In some embodiments, the baseline assessment may still be performed despite the pair being an established pair having interacted via a baseline assessment survey prior to the performance of Step304.

According to some embodiments, Step304can involve determining that a predetermined number of questions in a survey outstanding between a recipient-respondent pair are outstanding. For example, if a respondent has yet to answer 70% of questions put forth by a recipient, then Step304can be triggered which means that all of the questions of the outstanding survey are rendered “due,” which means that the respondent can be pinged or alerted to the outstanding nature of the survey and be requested to finish each question according to a set timing. In such embodiments, the outstanding survey can be viewed as the baseline assessment between that recipient-respondent pair.

In Step306, the results of the baseline assessment can be analyzed, and as a result, objectives therefrom can be identified and analyzed. According to some embodiments, the analysis of the baseline assessment can be performed by any type of machine learning (ML) or artificial intelligence (AI) model that can analyze survey data and determine the information provided therein, such as, but not limited to, classifiers, data mining models, neural networks, natural language processors (NLPs), and the like.

In some embodiments, a result of the analysis of the baseline assessment for each recipient-respondent pair can be realized as a determined score for the respondent. In some embodiments, scores can be determined based on the answered question, the unanswered questions, how long answers took to be provided, the content/context of the answer, and the like, or some combination thereof. In some embodiments, the scoring can be specific to a survey, set of surveys, a set of questions, a respondent(s), a recipient(s) and/or a recipient-respondent pair(s), and the like, or some combination thereof. According to some embodiments, scoring of the baseline assessment can be performed via the ML/AI models discussed above, among others, which can provide behavioral data for a respondent and/or their recipient-respondent pair.

In some embodiments, the baseline assessment analysis and scoring can enable the determination of the objectives for each recipient-respondent pair. In some embodiments, the objectives can include, but are not limited to, total utility value, fairness, recency, diversity and minimum number of questions per round.

As discussed below, these objectives can be leveraged to not only determine which questions to ask the respondent’s in the next round, but from which recipient’s the questions should originate from.

According to some embodiments, the total utility value objective corresponds to a number of total questions that have been answered by a respondent. In some embodiments, engine200can function to ensure this total utility value for each respondent is maximized so that each survey results in a high response rate.

In some embodiments, engine200can determine the utility value by determining a probability that a given respondent will actually answer a question from a given recipient, and multiply this probability by the number of questions given to that respondent. This product is an representation of the utility value.

In some embodiments, the fairness objective corresponds to a variance of the utility of the questions asked in a survey per round. This utility enables the questions to be fairly balanced, which can take into account biographical information, profile information, demographic information, geographic information, employment information (e.g., job title and level) and the like, when determining that questions are fairly balanced across each respondent. In some embodiments, engine200can function to ensure this fairness value has a minimized variance level to ensure common types of contextual questions across respondents. Effectively, engine200can provide an equitable fairness by ensuring the utility value is the same across users.

In some embodiments, the recency objective corresponds to a time since the recipient has heard from a respondent. This refers to how long it has taken a respondent to answer a survey that included a question(s) from a recipient. In some embodiments, engine200can function to ensure this recency value is minimized so that survey’s do not idle or become overdue. For example, reminders, notifications, alerts and/or incentives can be provided to respondents that have not answered questions beyond a threshold amount of time.

In some embodiments, the diversity objective corresponds to a number of different respondents that have provided a recipient with valid answer. In some embodiments, this objective can be a sub-part of the recency objective. In some embodiments, engine200can function to ensure that this diversity value is maximized so that more respondents are interacting with a recipient to provide a wider-breadth to the answers being provided to a recipient. In some embodiments, the diversity objective can also refer to a “delta-diversity”, that is, allocate respondents to recipients that have not provided such recipients valid feedback (at least within a threshold period of time), where feedback is considered valid if it addresses the question based on a contextual analysis that the answer’s context corresponds to the questions’ context.

In some embodiments, the minimum number of questions per round objective ensures that each respondent receives at least a minimum predetermined number n questions from their allocated recipient. For example, if respondent A is allocated to give feedback to recipient Y in round XX, engine200can allocate a minimum number of n questions between the Y-A pair (e.g., 2 questions, for example).

Having analyzed the identified objectives for each respondent, Process300proceeds from Step306to Step308where the objectives are optimized. According to some embodiments, Step308can involve engine200utilizing a solver in order to determine a “single verdict” (or representative feedback value) for each respondent. In some embodiments, the solver can be an implementation of any type of known or to be known optimization algorithm, such as, but not limited to, Annealer (which implements simulated annealing), HillClimber, (which implements a numerical analysis algorithm) ExhaustiveSwapper (which implements a bitwise swap operation), Greedy (which implements a greedy algorithm), and the like.

For example, Step308can involve using a greedy algorithm on each of the objectives for a respondent in order to optimize the results from the baseline assessment and determine a “single verdict” for that respondent that indicates how the respondent is expected to act in subsequent surveys (or rounds - such that the subsequent questions and/or recipients included in each round can be selected accordingly), as discussed below.

In Step310, question selection for each respondent is performed. In some embodiments, Step310involves determining which questions (e.g., types, topics and/or forms of questions) should be identified for each respondent, and this determination can be based on each respondent’s optimized objectives (from Step308).

According to some embodiments, each respondent’s score (from Step306) can be identified and weighted. In some embodiments, the weighting can be randomly applied (e.g., random weights per respondent) by applying a weighted randomness principle. In some embodiments, the weighting can be based on or directly correlate to the representative feedback value (or optimized objectives) of each respondent. For example, the objectives derived for each respondent provide an indication as to the nature of the respondent’s interaction with the survey, the questions included therein, and the recipients that are responsible for those questions. The optimization of these objectives can be leveraged so that the scoring of the respondent can be manipulated to further indicate how they will respond to like or dissimilar types of questions in the future. This, therefore, can be used to select questions to be included in surveys for the respondent’s moving forward.

In some embodiments, the representative feedback value can be weighted by a random value similar to the random weighting discussed above. In some embodiments, the scoring for a respondent can be weighted according to values of each objective (prior to or without them being optimized).

By way of a non-limiting example, questions that a respondent did not answer for a period of time (e.g., they are longer due) or questions that typically elicit the same type of response can be filtered out for that respondent.

Thus, Step 310′s question selection operation involves identifying questions that are more likely to elicit random or different responses from respondents (e.g., different responses per respondent). In some embodiments, the questions that are time sensitive, or have a “dueness” value attributed to them, can also be selected. According to some embodiments, the weighted scoring for a respondent can provide an indication as to which questions map to a respondent’s objectives (e.g., higher weighted scoring can indicate a likelihood that the questions will be answered and that they will elicit responses that are not expected and/or rudimentary, and therefore are compliant with the purposes of the survey).

In Step312, having identified which questions to select (or having selected the questions, as in Step310), engine200then performs question distribution. In some embodiments, Step312involves determines which recipients to have the questions originate from (e.g., who to send the survey). This determination can also be based on the scoring of the respondent and optimization of the objectives for the respondent as it provides an indication of which recipients each respondent is more likely to engage with (e.g., respond in a timely manner with engaging (or contextually relevant and descriptive) answers). According to some embodiments, engine200can identify the recipient(s) for a respondent based on a variety of factors, such as, but not limited to, for example, which recipient recently asked a question to the respondent, which recipient recently received a viable response from the respondent, how long were such questions “due,” what were the objective scores for the interactions between recipient-respondent interactions, and the like, or some combination thereof.

In Step314, having determined the questions (from Step310) and identified the recipients (from Step312), engine200can compile this information into an electronic survey and communicate an indication to a respondent(s) that a survey is being requested to be completed. In some embodiments, the compiled survey can correspond to the question budget and/or feedback cadence, as discussed above.

In some embodiments, the communication can comprise a link for the respondent to click to cause the survey to be opened. In some embodiments, the survey can be held in abeyance until the user opens the link, whereby the survey can be automatically generated at a time the respondent opens the survey (e.g., questions selected and populated, and recipient’s identified as per Steps310-312). In some embodiments, the compiled survey can be electronically communicated to the respondents in any electronic form (e.g., email, SMS, and the like). In some embodiments, each recipient that is selected (from Step312) can also receive a version, copy or indication related to the survey.

According to some embodiments, at the completion of Step314(e.g., sending the compiled survey to each respondent), Process300can recursively return to Step306, where the results of the survey can be analyzed in a similar manner as discussed above, whereby the objectives can be updated and Process300can function to prepare for subsequent survey rounds (according to the feedback cadence).

FIG.4is a block diagram illustrating a computing device400showing an example of a client or server device used in the various embodiments of the disclosure. Computing device400can be a representation of UE402, as mentioned above.

The computing device400may include more or fewer components than those shown inFIG.4, depending on the deployment or usage of the device400. For example, a server computing device, such as a rack-mounted server, may not include audio interfaces452, displays454, keypads456, illuminators458, haptic interfaces462, GPS receivers464, or cameras/sensors466. Some devices may include additional components not shown, such as graphics processing unit (GPU) devices, cryptographic co-processors, artificial intelligence (AI) accelerators, or other peripheral devices.

As shown inFIG.4, the device400includes a central processing unit (CPU)422in communication with a mass memory430via a bus424. The computing device400also includes one or more network interfaces450, an audio interface452, a display454, a keypad456, an illuminator458, an input/output interface460, a haptic interface462, an optional GPS receiver464(and/or an interchangeable or additional GNSS receiver) and a camera(s) or other optical, thermal, or electromagnetic sensors466. Device400can include one camera/sensor466or a plurality of cameras/sensors466. The positioning of the camera(s)/sensor(s)466on the device400can change per device400model, per device400capabilities, and the like, or some combination thereof.

In some embodiments, the CPU422may comprise a general-purpose CPU. The CPU422may comprise a single-core or multiple-core CPU. The CPU422may comprise a system-on-a-chip (SoC) or a similar embedded system. In some embodiments, a GPU may be used in place of, or in combination with, a CPU422. Mass memory430may comprise a dynamic random-access memory (DRAM) device, a static random-access memory device (SRAM), or a Flash (e.g., NAND Flash) memory device. In some embodiments, mass memory430may comprise a combination of such memory types. In one embodiment, the bus424may comprise a Peripheral Component Interconnect Express (PCIe) bus. In some embodiments, the bus424may comprise multiple busses instead of a single bus.

Mass memory430illustrates another example of computer storage media for the storage of information such as computer-readable instructions, data structures, program modules, or other data. Mass memory430stores a basic input/output system (“BIOS”)440for controlling the low-level operation of the computing device400. The mass memory also stores an operating system441for controlling the operation of the computing device400.

Applications442may include computer-executable instructions which, when executed by the computing device400, perform any of the methods (or portions of the methods) described previously in the description of the preceding Figures. In some embodiments, the software or programs implementing the method embodiments can be read from a hard disk drive (not illustrated) and temporarily stored in RAM432by CPU422. CPU422may then read the software or data from RAM432, process them, and store them to RAM432again.

The computing device400may optionally communicate with a base station (not shown) or directly with another computing device. Network interface450is sometimes known as a transceiver, transceiving device, or network interface card (NIC).

The audio interface452produces and receives audio signals such as the sound of a human voice. For example, the audio interface452may be coupled to a speaker and microphone (not shown) to enable telecommunication with others or generate an audio acknowledgment for some action. Display454may be a liquid crystal display (LCD), gas plasma, light-emitting diode (LED), or any other type of display used with a computing device. Display454may also include a touch-sensitive screen arranged to receive input from an object such as a stylus or a digit from a human hand.

Keypad456may comprise any input device arranged to receive input from a user. Illuminator458may provide a status indication or provide light.

The computing device400also comprises an input/output interface460for communicating with external devices, using communication technologies, such as USB, infrared, Bluetooth™, or the like. The haptic interface462provides tactile feedback to a user of the client device.

The optional GPS transceiver464can determine the physical coordinates of the computing device400on the surface of the Earth, which typically outputs a location as latitude and longitude values. GPS transceiver464can also employ other geo-positioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS, or the like, to further determine the physical location of the computing device400on the surface of the Earth. In one embodiment, however, the computing device400may communicate through other components, provide other information that may be employed to determine a physical location of the device, including, for example, a MAC address, IP address, or the like.

For the purposes of this disclosure the term “user”, “subscriber” “consumer” or “customer” should be understood to refer to a user of an application or applications as described herein and/or a consumer of data supplied by a data provider. By way of example, and not limitation, the term “user” or “subscriber” can refer to a person who receives data provided by the data or service provider over the Internet in a browser session, or can refer to an automated software application which receives the data and stores or processes the data.