Patent Publication Number: US-2021174307-A1

Title: System and methodologies for candidate analysis utilizing psychometric data and benchmarking

Description:
COPYRIGHT 
     A portion of the disclosure of this patent document contains material which is subject to (copyright or mask work) protection. The (copyright or mask work) owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all (copyright or mask work) rights whatsoever. 
     CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Application Ser. No. 62/933,744, filed Nov. 11, 2019, which is expressly incorporated by reference herein. 
     FIELD 
     The present disclosure relates to systems, components, and methodologies for analyzing the qualifications and capabilities of one or more employment candidates. 
     BACKGROUND 
     Conventionally, many organizations perform executive search services that include researching the availability of suitable candidates. These search firms may act as intermediaries to identify potential candidates and/or analyze candidates&#39; interest, qualifications or expertise. 
     Conventionally, both identification of potential candidates and hiring decisions have been based on incomplete data and/or personal subjective opinion (e.g., self-written resumes and personal interviews). Nevertheless, certain personality traits that play a significant role in determining an individual&#39;s success in a given task/job may be intangible and not easily identified in such situations. 
     SUMMARY 
     As a result, in such situations, analysis and hiring decisions should be based on more complete, objective data pertaining to candidates&#39; interests, qualifications, skills, experience and expertise as well as identification of candidate personality traits that enable success in a particular job. 
     In accordance with at least some embodiments, such personality traits may be validated using one or more psychometric tools to identify and measure the presence of such traits in employment candidates. 
     In accordance with at least some embodiments, systems, components and methods are provided for analyzing the interests, skills, experience, qualifications and capabilities of one or more employment candidates in a manner that enables matching employment candidates to specific job roles by utilizing a benchmarking process that incorporates these variables. In accordance with at least some embodiments, a benchmarking process is also used to further improve the ability to accurately match employment candidates to specific job roles. In accordance with at least some embodiments, a system and methodologies formulate an objective benchmark utilizing a reverse psychometric model. 
     In accordance with at least some embodiments, the benchmarking process and formulation may include psychometric scores (i.e., psychometrics), as well as scoring data pertaining to candidate fit against other relevant job attributes (e.g., screening questions, experience, skills test scores, cultural preference, etc.), wherein all of the data are combined into a single “fit score” that provides a holistic and representative, quantitative data regarding a candidates overall fit for a particular role. 
     Additional features of the present disclosure will become apparent to those skilled in the art upon consideration of illustrative embodiments exemplifying the best mode of carrying out the disclosure as presently perceived. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
       The detailed description particularly refers to the accompanying figures in which: 
         FIG. 1  is structured flow diagram illustrating a benchmark creation process provided in accordance with the disclosed invention. 
         FIG. 2  is a flow chart illustrating an example of a model building process provided in accordance with the disclosed invention. 
         FIG. 3A  is an example of an initial assessment of a Psychometric Benchmark Instrument questionnaire provided in accordance with the disclosed invention. 
         FIG. 3B  is an example of a question of the Psychometric Benchmark Instrument questionnaire provided in accordance with the disclosed invention. 
         FIG. 3C  is a further example of a question of the Psychometric Benchmark Instrument questionnaire provided in accordance with the disclosed invention. 
         FIG. 4  is an example of an Applicant Tracking System dashboard provided in accordance with the disclosed invention. 
         FIG. 5  is an example of the Applicant Tracking System dashboard provided in accordance with the disclosed invention. 
     
    
    
     DETAILED DESCRIPTION 
     The figures and descriptions provided herein may have been simplified to illustrate aspects that are relevant for a clear understanding of the herein described devices, systems, and methods, while eliminating, for the purpose of clarity, other aspects that may be found in typical devices, systems, and methods. Those of ordinary skill may recognize that other elements and/or operations may be desirable and/or necessary to implement the devices, systems, and methods described herein. Because such elements and operations are well known in the art, and because they do not facilitate a better understanding of the present disclosure, a discussion of such elements and operations may not be provided herein. However, the present disclosure is deemed to inherently include all such elements, variations, and modifications to the described aspects that would be known to those of ordinary skill in the art. 
     In accordance with at least some embodiments, such personality traits may be validated using one or more psychometric tools to identify and measure the presence of one or more character or personality traits in employment candidates so enable matching employment candidates to specific job roles utilizing psychometric data. 
     Conventionally, hiring decisions have been based on incomplete data and/or personal subjective opinion (e.g., self-written resumes and personal interviews). Nevertheless, certain personality traits, experiences, skills and interests that play a significant role in determining an individual&#39;s success in a given task/job are sometimes intangible and not easily identified in such situations. 
     Without analysis of those traits being incorporated into the hiring decision, the accuracy of the placement of an individual in a role is often times low, leading to poor performance and job dissatisfaction. When these traits are identified and consideration of the traits is incorporated into the hiring decision, overall job performance and match can be improved. 
     Accordingly, disclosed embodiments provide a formalized mechanism by which hiring managers can incorporate analysis of these psychometric traits into the hiring decision. 
     In accordance with disclosed embodiments, a system and methodologies are provided to analyze and rank employment candidates in such a way that predicts an individual candidate&#39;s fit to a particular job. However, effective determination and use of such personality trait data requires validated psychometric tools to identify and measure that data. The defining feature of this approach is its empirical foundation. The term “psychometric” simply refers to the quantitative assessment of psychological states/abilities. Contemporary psychometric tests consist of varied subtests that tap diverse aspects of a defined construct of traits and characteristics. For example, scales may include subtests that sample a more specific set of traits (e.g., extraversion, empathy, pro-activeness, detail-orientation) and require the examinee to rank order statements (i.e., proxy statements) that represent the underlying constructs in a way that prevents the examinee from arbitrarily influencing or skewing the results. The resulting rankings represent an objective representation of their behavioral preferences and tendencies. 
     At the heart of psychometric testing lies norm referencing (Sattler 1992). Norm referenced tests are developed by administering items in a standardized manner to a representative sample of the population in question. The norm sample is considered ‘representative’ insofar as it is stratified within different groups for variables that might influence performance differentially (e.g., gender, geographic region, ethnic status, community size, etc.). Scores are scaled such that each individual&#39;s derived score represents a relative standing within the norm or standardization group. In this sense, psychometric testing is an empirical endeavor in its purest sense. 
     At least some disclosed embodiments utilize an artificial intelligence-driven automated comparison to a benchmark to improve objectivity and accuracy of analysis. 
     In accordance with at least one implementation, this may be accomplished by capturing personality trait data for one or more personality traits from a targeted population, studying that data to determine an ideal “benchmark” for the one or more traits and automatically comparing unlimited candidates against that benchmark to arrive at a comparative job/fit score ranging across a customized continuum, e.g., from excellent, very good, average or poor, or other similar categorizations of employment candidate job fit. 
     A particular technical utility is provided by the disclosed embodiment&#39;s use of a psychometric approach to building benchmarks as a result of the nature of psychometrics to begin with. Such an approach to building benchmarks takes advantage of the reliability and validity predictors of performance (Sattler 1992). The term “reliability” refers to consistency of measurement; the more reliable a measure, the less error involved in estimates derived from it. When done well, psychometric tests boast extremely high internal reliability (the degree to which each component score of the test correlates with the full test score) and short-term ‘test-retest’ reliability (an index of stability derived by administering the test to the same group of individuals more than once). The validity of these tests, too, has proven to be strong. The term “validity” refers to the extent to which a test measures what it was designed to measure. 
     Thus, in implementation, disclosed embodiments are based on examination of a single role/job for which a benchmark is to be formulated. Benchmarks may be formulated, or established, through a three-stage process which is a combination of subject matter expert feedback, statistical comparisons and use of a Psychometric Benchmark Instrument (“PBI”). 
     For example, in a subject matter expert feedback stage, thought leaders and practitioners trained in the psychometric profiles, and with industry experience in the role, may provide input on which personality traits are most important for success in a given role/job. These experts may be both proficient in the analysis of personality profiles and potentially have experience in a particular industry and/or job/role. To some extent, this phase receives subjective input because analysis may be performed based on subjective analysis of one or more personality traits and how those traits affect job/role/task performance. 
     Thereafter, statistical comparisons of personality traits may be performed from a cohort of top and bottom performers for a job/role/task. It should be appreciated that this phase of the analysis provides an objective input. Data for analysis may be obtained from personality profile data from a database of individuals in a particular role; thereafter, statistical analysis may be performed to determine statistically significant correlations between traits and performance. Optionally, an adjustment may be made for distribution in the population to normalize the benchmark so it is neither overly punitive (i.e., too small a percentage of the population would score high) nor too forgiving (i.e., too large a percentage of the population would score high). 
     Subsequently, the PBI may be used to enable key stakeholders from the particular industry being analyzed to provide input. For example, this phase may include individuals excelling in a particular job/role and/or managers of those individuals to complete the PBI to generate psychometric input). In at least one implementation, the PBI stakeholders may be tasked with rank ordering a series of statements that represent the core dimensions of a personality profile. Proxy statements may be used to hide core dimensions of the assessment and substitute every day observable behaviors so as to reduce subjective bias. These rankings may be scored using computer algorithms that contribute significantly to determining final scores in the benchmark. Such final scores may be used to make predictive statements about the candidate&#39;s fit and future performance in the particular job/role. 
       FIGS. 3A-3C  show an example of a PBI questionnaire that may be sent to stakeholders. The questionnaire may be electronically delivered to the stakeholders such that they may fill out the initial assessment  160  and the questions  170  electronically. As can be seen in  FIG. 3A , an initial assessment  160  will ask for basic information regarding the job. The initial assessment  160  may include fillable boxes for information such as “Job Title”, who the employee would report to (“Reports To”), and may allow for a “Job Description” to be attached to the assessment  160 . The assessment  160  may also allow for additional “Consultant Notes” to be added.  FIGS. 3B and 3C  show examples of questions  170  that the stakeholders may answer. For example,  FIG. 3B  shows a question  172  regarding desirable traits of the job. In another column  174 , the question  170  lists traits that the stakeholder may rank according to relevance to the specific job role. In a further column  176 , the traits from the previous column  174  will appear ranked based on how the stakeholder&#39;s ranking. In order to rank the traits, the stakeholder may click and drag the traits around the computer window until an acceptable rank is achieved.  FIG. 3C  shows a similar question  170 , but involving a ranking behaviors as opposed to traits. 
     For example, a benchmark may assess an individual candidate&#39;s fit, and make predictive statements, in a job/role in, for example, one of various areas. For example, analysis may be performed based on task and duty fit analysis based on the DISC theory created by Dr. William Marston at Harvard, so as to set an ideal score for the core four dimensions of behavioral fit: decisiveness, interactiveness, stability and conscientiousness. 
     For example, in decisiveness (problems), scoring may be performed to determine how decisive and agile does the employee need to be in order to make decisions in the role. More decisive roles (e.g., sales, law enforcement, field military operations) may require a more intuitive and quick response time to making important decisions with less time and information. In these roles, there may be less time and information available to make decisions. In less decisive, or more deliberate, roles (e.g., quality control, accounting, legal) the ideal employee may take more time to make more controlled and deliberate decisions, with more information gathering and less room for mistakes or corrections once the decision is acted upon. 
     In interactiveness (people), scoring may be performed to determine how interactive and social does the employee need to be in order to effectively execute their job responsibilities. In highly interactive roles, the employee may need to be more extraverted and outgoing (e.g., sales, leadership, persuasive tasks, public or customer-facing support roles). They may need to be naturally comfortable meeting new people and interacting with them in an effective manner. In less interactive roles (e.g., back-office, individual contributor roles) there may be less of a need for highly developed social skills. The individual can deliver the majority of their deliverables by working independently, remotely or with less interaction with a team, the pubic or customers. 
     In stability (pace), scoring may be performed to determine how agile or structured does the employee need to be in order to keep up with the pace of the work. In highly structured roles (e.g., auditors, analysts, forensics) the employee may have to take a slower pace, operating in a more structured and rigid environment and move at a slower pace. In less structured roles (e.g., sales, art, entertainment, EMS), there may be less stability, less predictability and the employee may need to be comfortable operating with more unknowns and less certainty. 
     In conscientiousness (procedures), scoring may be performed to determine how detailed and accurate does the employee need to be in order to fulfill their responsibilities. In highly detailed roles (e.g., accounting, legal, administrative), the tasks may require a high degree of accuracy and attention to detail. The role may center around the minutia of the equation. In less detailed roles (e.g., sales, entrepreneurism, creative work), there may be a lesser demand for such attention to detail and “in-the-box” thinking. The ideal employee may be free to focus on the higher-order “big-picture” aspects of the work, rather than the detailed specifics. 
     In engagement and motivation fit, scoring may be performed to determine how, for example, based on Eduard Spranger&#39;s model of motivation and engagement, to set an ideal level of motivation on each of one more dimensions, for example:
         Aesthetic: How motivated is the employee to achieve balance and harmony in their life, and the environment in which they work? To what degree does the work or culture reward an individual&#39;s motivation to achieve such a balance, or create balance in world as a result of the work they do (e.g., work-life balance, artistic expression, green initiatives, harmoniousness among co-workers)?   Economic: How motivated is the employee to drive a higher Return on Investment of available resources? To what degree does the work and culture reward an individual&#39;s motivation to grow resources, increase value, develop revenue as a result of the work they do (e.g., driving top-line revenue, increasing shareholder value, earning more money)?   Individualistic: How motivated is the employee to be independent and express themselves in unique ways? To what degree does the work and culture reward an individual&#39;s motivation to be independent, determine their own way or process of things, control their own destiny and create new ways of doing things (e.g., creating new approaches, pioneering experimental avenues, expressing uniqueness and individuality)?   Political: How motivated is the employee to take the lead and have authority and responsibility for the task or others? To what degree does the work or culture reward an individual&#39;s motivation to have responsibility for others, be followed and relied upon, be in charge or be giving ownership of the deliverable (e.g., Be the leader, coach others, guide the sale)?   Altruistic: How motivated is the employee to support others and give selflessly in order to benefit them, not themselves? To what degree does the work and culture reward an individual&#39;s motivation to care for the needs of others, support them, give without expectation of reward (e.g., supporting the team, donating their time, sacrificing for the good of others or the world)?   Regulatory: How motivated is the employee to ensure compliance with regulation? To what degree does the work and culture reward an individual&#39;s motivation to ensure rules exist and are followed, structures are defined and policies kept (e.g., writing guidelines, creating and enforcing rules and practices, organizing and structuring activities)?   Theoretical: How motivated is the employee to constantly learn and acquire new knowledge? To what degree does the work and culture reward an individual&#39;s motivation to grow their knowledge base, explore new topics and gain new understanding (e.g., research studies, theoretical experimentation, educational endeavors)?   Once a benchmark is created it is considered “provisional” until it has been used by sufficient employers, who offer feedback and aggregate data, to achieve a level of field validity where we can mark the benchmark as “validated” and then make it available for the public client-base. During this feedback and refinement process employees offer input that helps adjust the benchmark to a final degree of accuracy.       

       FIG. 1  illustrates an example of a structured flow diagram illustrating a benchmark creation process provided in accordance with disclosed embodiments. As shown in  FIG. 1 , operations may begin at  105  and control may proceed to either  110  to perform a reactive external build of a benchmark or  115  to perform a reactive internal benchmark build. At  110 , reactive external builds may be performed based on three options: trained business consultants (e.g., opinion, client stakeholders (benchmark instrument)) or client employees (e.g., profiling live 700+ people). At  115 , reactive internal builds may be performed by personnel that examine a particular job, researches particulars of that job and assigns benchmark scores based on that examination and research. 
     Following completion of the reactive build options at  110 - 115 , control proceeds to a strawman operation is performed to verification/validation of hypothetical benchmark scores at  120 . Thereafter, the data generated by that operation are available for use by subsequent reactive external builds at  110 , subsequent proactive external builds performed at  130  (which may use a pool of consultants that may offer benchmark scores), and subsequent statistical validation at  125  (which may involve accessing cohort scores for analysis and validation, e.g., sample size of 2,000+ profiles). Control may then proceed to  135 , at which generated benchmark scores are stored in the finished provisional benchmark library. Thereafter, the provisional benchmark may be validated at  140 , e.g., by validating, for example, fifty use cases. Control may then proceed to  145  for storage in one or more validated benchmark libraries, e.g., a private, client specific library or a public library (optionally including public benchmark libraries in resources with validated benchmark libraries being useful for complete and live job analysis. Control then proceeds to  150 , at which operations end. 
     In accordance with at least some embodiments, a system and methodologies formulate an objective benchmark utilizing a reverse psychometric model.  FIG. 2  is a flow chart illustrating an example of a model building process  200  provided in accordance with the disclosed invention. As shown in  FIG. 2 , the model building process  200  begins at operation  210  wherein meta inputs are provided via input or access as known in the art. The meta inputs may include stakeholder input data  212  and/or job description data  214 . Following completion of the operations at  210 , control proceeds to operation  220 , which pre-processes the meta data provided, in particular pre-processing the job description data  214 . The job description data  214  may pre-processed to perform sample data discovery  222 . Further, in operation  220 , the sample data discovery  222  may be utilized to generate linked data  224 . 
     The generation of the linked data  224  may be executed via program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. The following pseudo code provides an example of the execution of the generation of the linked data  224 : 
     
       
         
           
               
             
               
                   
               
             
            
               
                 procedure generate_linked_employee_record (employee E, job J, assessments A): 
               
            
           
           
               
               
            
               
                   
                 AnonymousEmployeeRecord = Object( ) 
               
               
                   
                 # Attach employee&#39;s raw performance score, which is arbitrary per company 
               
               
                   
                 AnonymousEmployeeRecord.raw_peformance_score =E.raw_peformance_score 
               
               
                   
                 # Attach raw values for each complete assessment 
               
               
                   
                 for each assessment in A: 
               
            
           
           
               
               
            
               
                   
                 if assessment.complete: 
               
            
           
           
               
               
            
               
                   
                 for each subscore in assessment.scores: 
               
            
           
           
               
               
            
               
                   
                 AnonymousEmployeeRecord.set(subscore.name, subscore.value) 
               
            
           
           
               
               
            
               
                   
                 # Only attach proficiency levels for relevant skills and competencies per job 
               
               
                   
                 for skill in J.skills: 
               
            
           
           
               
               
            
               
                   
                 if skill in E.skills: 
               
            
           
           
               
               
            
               
                   
                 AnonymousEmployeeRecord.set(skill.name, 
               
               
                   
                 E.skills.get(skill).proficiency_level) 
               
            
           
           
               
               
            
               
                   
                 else: 
               
            
           
           
               
               
            
               
                   
                 AnonymousEmployeeRecord.set(skill.name, None) 
               
            
           
           
               
               
            
               
                   
                 return AnonymousEmployeeRecord 
               
               
                   
                   
               
            
           
         
       
     
     Thereafter, control proceeds to operation  230 , during which modeling inputs is performed, e.g., the stakeholder input  212  may be used as modeling inputs to generate strawman initial parameters  232 . Likewise, at  230 , the linked data  224  from operation  220  of the data pre-processing may be used as modeling inputs for psychometric sample data  234  and raw performance data generation  236 . 
     Thereafter, the model building process  200  may proceed to operation  240 , wherein further data pre-processing is performed e.g., the psychometric sample data  234  from operation  230  of the modeling inputs may be used to generate test/train/validation data sets  242 . Also at operation  240 , the raw performance data  236  may be used to generate normalized rank-order performance data (by quartile)  244 . Thereafter, the model building process  200  may perform model training at operation  250 , wherein the strawman initial parameters  232  from operation  230  may be used for the model training at operation  250 , wherein machine learning classification algorithms may be trained  252  using various algorithms, e.g., Support Vector Machines (SVMs), Neural Network, etc. The model training at  250  may also be informed by test/train/validation sets generated  242  in the further data pre-processing of operation four  240  based on psychometric sample data  234  and normalized rank-order performance data (generated, for example, by quartile)  244  based on the raw performance data modeling inputs  236 . 
     The generation of the normalized rank-order performance data  244  may be executed via program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. The following pseudo code provides an example of the execution of the generation of the normalized rank-order performance data  244 : 
     
       
         
           
               
             
               
                   
               
             
            
               
                 procedure normalize_partition_performance_data (companies C): 
               
            
           
           
               
               
            
               
                   
                 TopPerformers = List( ) 
               
               
                   
                 BottomPerformers = List( ) 
               
               
                   
                 for all company in C: 
               
            
           
           
               
               
            
               
                   
                 S = List( ) 
               
               
                   
                 while company.raw_peformance_data is not empty: 
               
            
           
           
               
               
            
               
                   
                 AnonymousEmployeeRecord = company.raw_peformance_data.pop( ) 
               
               
                   
                 S.push(AnonymousEmployeeRecord) 
               
            
           
           
               
               
            
               
                   
                 SampleSize = S.size( ) 
               
               
                   
                 S.sort(key=‘raw_peformance_score’) 
               
               
                   
                 # Add top quartile based on Company&#39;s performance score 
               
               
                   
                 TopQuartileIndex = int(SampleSize * 0.25) 
               
               
                   
                 TopPerformers.append(S[0 : TopQuartileIndex]) 
               
               
                   
                 # Add bottom quartile based on Company&#39;s performance score 
               
               
                   
                 BottomQuartileIndex = int(SampleSize * 0.75) 
               
               
                   
                 BottomPerformers.append(S[BottomQuartileIndex : SampleSize−1]) 
               
            
           
           
               
               
            
               
                   
                  return TopPerformers, BottomPerformers 
               
               
                   
                   
               
            
           
         
       
     
     In accordance with at least some disclosed embodiments, the model building process  200  may further proceed to a further modelling input operation  260 , wherein, based on the trained machine learning classification algorithms  252 , several algorithms are assessed and the most performant algorithm is selected as  262 , as measured by the Receiver Operating Characteristics (ROC). Of particular note, the second data pre-processing stage of operation  240  illustrated in  FIG. 2  includes transformation of any performance data into rank-ordered data  244 , whereby further classification of the top and bottom quartiles of data may be performed as a binary outcome of job fit. 
     The training of machine learning classification algorithms  252  may be executed via program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. The following pseudo code provides an example of the execution of the generation of a list of trained and validated machine learning classification algorithms  252 : 
     
       
         
           
               
             
               
                   
               
             
            
               
                 procedure train_predictive_models (attributes Strawman, employees TopPerformers, 
               
               
                 employees BottomPerformers): 
               
            
           
           
               
               
            
               
                   
                 # Build various training models from library of Machine Learning Classification 
               
            
           
           
               
            
               
                 Algorithms 
               
            
           
           
               
               
            
               
                   
                 M = List( ) 
               
               
                   
                 for all learning_model in ClassifcationAlgorithms: 
               
            
           
           
               
               
            
               
                   
                 # Initialize each model with expert chosen Strawman parameters 
               
               
                   
                 training_model = learning_model(bootstrap_parameters=Strawman) 
               
               
                   
                 # Train each model on binary outcomes of top and bottom performers 
               
               
                   
                 trained_model = training_model.fit(positive_case=TopPerformers.training_data, 
               
            
           
           
               
            
               
                 negative_case=BottomPerformers.training_data) 
               
            
           
           
               
               
            
               
                   
                 # Validate trained model against test data 
               
               
                   
                 validation_scores = cross_val_score(trained_model, 
               
            
           
           
               
            
               
                 positive_case=TopPerformers.testing_data, 
               
               
                 negative_case=BottomPerformers.testing_data) 
               
            
           
           
               
               
            
               
                   
                 # Select models which are better than random for further review 
               
               
                   
                 if validation_scores.mean( ) &gt; 0.5: 
               
            
           
           
               
               
            
               
                   
                 M.push(trained_model) 
               
            
           
           
               
            
               
                 # Return list of validated machine learning classification algorithms 
               
               
                 return M 
               
               
                   
               
            
           
         
       
     
     The determination of the optimal machine learning classification model  262  may be executed via program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. The following pseudo code provides an example of the execution to determine the optimal machine learning classification model from a list of validated models  262 : 
     
       
         
           
               
             
               
                   
               
             
            
               
                 procedure determine_optimal_model(models M): 
               
            
           
           
               
               
            
               
                   
                 # Determine the highest qualify validated model using Receiver 
               
               
                   
                 Operating Characteristics 
               
               
                   
                 MaxAUC = 0 
               
               
                   
                 OptimalModel = None 
               
               
                   
                 for all validated_model in M: 
               
            
           
           
               
               
            
               
                   
                 if validated_model.auc &gt; MaxAUC: 
               
            
           
           
               
               
            
               
                   
                 MaxAUC = validated_model.auc 
               
               
                   
                 OptimalModel = validated_model 
               
            
           
           
               
               
            
               
                   
                 # Return the final OptimalModel which contains human readable 
               
            
           
           
               
            
               
                 features and can score job seekers by fit 
               
            
           
           
               
               
            
               
                   
                 return OptimalModel 
               
               
                   
                   
               
            
           
         
       
     
     The machine learning classification algorithms  252  may utilize feedback from feedback sources in order to periodically receive feedback and further revise and refine the models discussed above. The feedback sources may include, for example, the subject matter expert feedback, statistical comparisons and use of the PBI discussed above or other sources. The machine learning classification algorithms  252  may result in improved models which may more accurately assess a candidate&#39;s personality fit for the job. 
     In at least some embodiments, the system may periodically retrain the predictive psychometric models discussed above using additional real-world binary outcome data provided by hiring managers through the use of the ATS system stages  310  discussed below. Specifically, hiring managers may be surveyed to verify the performance of applicants placed in the “Hired” stage  320  and determine new hires who are top and bottom performers. This additional data may be then be used to produce an improved psychometric model through the same process used to build the initial model, e.g. model building process  200 . 
     In accordance with at least one embodiment, the disclosed embodiments may enable the ability to dynamically track employment candidate scores in association with a particular job/role. Such an implementation may enable dynamically scoring employment candidates through a “guided iterative input” process for a hiring manager. 
     Conventional Applicant Tracking System (ATS) (example of an ATS dashboard  300  is shown in  FIGS. 4 and 5 ) software is known for assisting hiring managers electronically organize and manage the flow of applicants for a given job through various stages of the hiring process. Its primary objective is to organize large numbers of applicants into stages or phases of the selection process. 
     Not unlike Customer Relationship Management (CRM) software, which is designed to coordinate and track sales and customer information, an ATS achieves the same function only for employment applicants. It is conventionally known to include additional features like resume sorting and screening, where the software makes some determination of fit based strictly on keyword searches in the resume (job titles, qualifications, schools, experience, etc.). 
     However, recently Applicants have learned to skew the accuracy of such ranking systems by including key words in their resume designed to help them rank higher (e.g., including the term “Search Engine Optimization (SEO)” in web-based marketing). Thus, the principal function of conventional ATSs has been strictly to help organize a company&#39;s recruitment efforts, not assist the hiring manager in deciding who the optimal candidate actually is. Thus, conventional ATSs improve efficiency not accuracy. 
     To the contrary, in accordance with at least some disclosed embodiments, an ATS is provided that tracks employment candidates but also contributes to the actual hiring decision by screening and ranking on a plurality of levels of qualification. 
     In implementation, this may involve moving candidates through a defined workflow wherein they are scored against a benchmark standard in each stage of the process. Such an implementation enables the ability to separate the hiring and benchmark comparison process down into smaller, more easily discernable actions, that score each candidate in a variety of functional areas relevant to a particular job. In doing so, the ATS provided by the disclosed embodiments goes beyond simply a means of tracking, to a means of screening and ranking and a decision-support tool designed to help the hiring manager make a better hiring decision. 
     The system operates by dividing the recruitment process into various and a plurality of customizable “stages”  310 . Each stage of the plurality of stages  310  is sequential and the flow is configured to assess a candidate&#39;s fit in each stage of the plurality of stages  310  and then move them to the next, to be evaluated against that stage&#39;s unique qualification. In each stage, for example, a candidate may receive a score on a 1-5 scale; as they progress through the plurality of stages  310 , the scores for each stage may be aggregated, which effects the overall ranking of any individual candidate relative to the other candidates. 
     By the time a candidate reaches a final stage of the plurality of stages  310 , the data generated by the ATS and displayed on the ATS dashboard  300  (an example being illustrated in  FIGS. 4 and 5 ) have enabled the hiring manager to make numerous “micro-evaluations.” In the example shown in  FIGS. 4 and 5 , one stage of the recruitment process may include a personality fit  312  assessment. Such micro-evaluations to be performed by the hiring manager may include evaluating a candidate based on their personality fit for the position. These micro-evaluations are designed to break the overall lengthy and burdensome process of hiring into easy-to-digest, quick judgments. The ATS designed in accordance with the disclosed embodiments may track these judgments (assessments) for the hiring manager and automatically compare all candidates against each other relative to their overall rank. The ATS may also aggregate the personality fit for each candidate so as to indicate the total number of candidates who fall into categories of the personality fit assessment  312 , such as “Excellent”, “Very Good”, “Average”, “Poor”, and “Incomplete”, as shown in  FIGS. 4 and 5 . Moreover, as shown in  FIG. 3 , a listing of the candidates may be accessed via a drop-down menu  330  or via a main candidate column including an “Applied” stage  314  including more detailing information of each candidate. 
     For example, consider the following process. Candidate Doe applies for a job and is asked to complete a personality profile assessment  312  associated with a first stage  314  of the plurality of stages  310 , which could be an “Applied” stage  314 , in the ATS in order to ascertain a personality fit. The software compares the results of that personality profile against a benchmark formulated as explained above and sorts stage one candidates as one of an: “Excellent”, “Very Good”, “Average”, “Poor”, fit for the job (based on the personality assessment  312 ), as shown in  FIGS. 4 and 5 . 
     Subsequently, the hiring manager is provided with data on the ATS dashboard  300  so that she can examine candidates in a subsequent stage of the plurality of stages  310  (ideally based on top-down ranking, and moves them to stage two, where she examines the candidates resume). On the ATS Dashboard  300  screen may be displayed a ranking bar that allows the hiring manager to rate the candidate&#39;s resume on a 1-5 scale. For example, as shown in  FIG. 5 , the stages  310  may include a “Qualified” stage  316 . 
     Thereafter, once the ATS registers a new score in stage two  316  of the plurality of stages  310  (i.e., resume rating as shown as a “Qualified” stage  316  in  FIG. 5 ), the system may automatically incorporate that data into the overall ranking and adjust the individual candidates ranking among all other candidates. 
     Subsequently, the ATS system may be configured to enable the hiring manager to move the candidate to stage three  318  of the plurality of stages  310 , shown as an “Interviewing” stage  318  in  FIG. 5 . Based on the interview, the hiring manager can rate the candidate on the familiar 1-5 scale. As the system registers a new score in stage three  318 , it is configured to incorporate that data into the overall ranking and adjust the individual candidates ranking among all other candidates. 
     At each stage  310 , the system incorporates the rating for that stage  310  and automatically updates the cumulative rankings of all candidates. For example, after the vetting process is complete, the hiring manager may move the candidates to a “Hired” stage  320  or a “Disqualified” stage  322 . The ATS system also enables the hiring manager to move candidates through however many stages  310  that may exist (the system may come with a predetermined set of stages  310  but can be customized by the hiring manager to fit their unique hiring process). The hiring manager may add additional stages via an “Add a new stage” button  324  on the dashboard  300 . 
     The unique aspect of this implementation of the disclosed embodiments modifies conventional technologies and capabilities to provide a single dynamic process that integrates numerous functions into one feedback system that support a more efficient and effective recruiting and hiring program. 
     Generated rankings may serve as predictions of candidate success in a position and may be significantly more accurate than traditional “key words” based screening because the presently used algorithms match not only “job” requirements and candidate skills, but are also based on historically successful candidate characteristics provided by the psychometric data. The attributes considered may include educational background, career trajectory, caliber of research produced and business outcomes generated by successful candidates. These factors may be assigned weights based on their predictive value and incorporated into the algorithm. 
     In accordance with the disclosed embodiments, functionality may be theoretically considered to be implemented in one or more of a backend, middleware, and a frontend. Moreover, in such an implementation, one or more of the system functionality may be implemented on one or more servers coupled to one or more public and/or private communication networks, e.g., the Internet. As a result, it should be appreciated that the functionality disclosed herein and interaction with the system subcomponents may be accessed via one or more computers, e.g., a mobile device such as a mobile smartphone, a PDA, a personal computer, a multimedia computer, a laptop, and the like. Additionally, or alternatively, the functionality disclosed herein may be implemented in whole or in part as a software application, e.g., a mobile application, which is a software program or application designed to run on computers such as smartphones, tablet computers and other mobile devices. 
     The software application may communicate with one or more databases running on such servers to provide to enable storage of uploaded data from candidates as well as storage of datasets for download to candidates as part of the skills testing subcomponent one or more communication networks. Such networks may include any type of communication network, such as a cellular communication network, including but not limited to various generations of cellular network technology (2G-5G) Global System for Mobile Communications (GSM), Wideband Code Division Multiplex Access (WCDMA), Code Division Multiplex Access (CDMA), or Time Division Multiplex Access (TDMA), General Packet Radio Services (GPRS), Universal Mobile Telephone System (UMTS). Additionally, such networks can also be implemented as a combination of two or more technologies i.e., a hybrid network. Further, the network(s) may also include generic Internet access using any transport methods. 
     The functionality disclosed herein may be implemented in various configurations using computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise physical storage and/or memory media such as RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. 
     Accordingly, although not illustrated, it should be understood that the functionality may be implemented using a server that provides or includes a processor connected to a user interface, computer readable memory and/or other data storage and a display and/or other output device. Computer executable instructions and data used by a processor may be stored in the computer readable memory included in the server or implemented with any combination of read only memory modules or random access memory modules, optionally including both volatile and nonvolatile memory. 
     While this invention has been described in conjunction with the specific embodiments outlined above, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the various embodiments of the invention, as set forth above, are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the invention. 
     Further, it should be understood that the functionality described in connection with various described subcomponents of various invention embodiments may be combined or separated from one another in such a way that the architecture of the invention is somewhat different than what is expressly disclosed herein. Moreover, it should be understood that, unless otherwise specified, there is no essential requirement that methodology operations be performed in the illustrated order; therefore, one of ordinary skill in the art would recognize that some operations may be performed in one or more alternative order and/or simultaneously. 
     Although certain illustrative embodiments have been described in detail above, variations and modifications exist within the scope and spirit of this disclosure as described and as defined in the claims included in this application. Moreover, it should be understood that any recitation of examples of pseudo code are mere examples of how the disclosed functionality may be executed via program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Thus, these recitations of pseudo code are not limiting to the disclosed embodiments but merely enabling of the disclosed inventive concepts.