Accountability	4; 22	Have  the project AE and the BAU process owner been made aware of, and signed off, the RAI risk assessment and mitigation steps?	Yes	"Project AE: Name and Designation - Watson Stewart -Head of Talent Solutions

PO: Name and Designation - Leanne Newton, Head of Early Careers and Talent Accelerators

Approved at: NFRC/PgSC etc.

Approval Documentation: PED/CR etc."	"Provide the following details:
1) Name and designation of the Project's AE
2) Name and designation of the Process Owner
2) How was the approval obtained (e.g. PED / CR)
3) Which forum/committee has approved (e.g. NFRC/PgSC)"
Accountability	3a; 3b	Have the trade-offs around using AI/ ML (vs alternatives) been considered as part of the above? Provide justification.	Yes	Pymetrics use of AI/ML allows us to generate an objective predictive model of fit to the role, rather than relying on a perceived notion of what would make an employee excel at the job. This shift from a subjective to an objective analysis also contributes to the overall reduction of conscious and unconscious bias.​ The use of machine learning also allows for insight into both the recall and accuracy of the model prior to deployment, an advantage not found in traditional assessments. 	Explain the incremental benefit from using AI/ML in place of traditional statistical models or rule based decision-making.
Accountability	22	Will the relevant Country Management and Board Committees be informed of this use case? (note that the actual 'informing' will be done periodically, and is not a pre-requisite for approval)	Yes		"For use cases impacting SG and HK, RAI Council provides periodic updates to the respective DGCs on all use cases regardless of materiality.

For material use cases, the respective DGCs are expected to include them in their periodic updates to ERC for noting"
Accountability	22	Has the relevant non-financial risk committee (or equivalent) provided its approval on issues not directly covered under the RAI Standards (e.g., broader operational risk)?*	Not Applicable		"Provide the following details if response is Yes:
1) Which committee has approved?
2) What are the issues that are being addressed?"
					
Data Suitability	6; 7c	Is the dataset used to train the algorithm:		See below re incumbent datasets	
Data Suitability	6	"1) representative of the relevant population?
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	All pymetrics assessments are thoroughly validated to ensure they are job-relevant, valid and reliable. The Project team have worked alongside Pymetrics to ensure an adequate number of incumbents have undertaken the assessment in order to provide the necessary data to train the AI. Not only this, but those incumbents selected to take part in the data collection process were also chosen based on their success in role against the requirements of the job function that those undertaking the assessment will be applying to, i.e. relevant people data is being used in order to inform the model.  In addition, the skills requirements of the function will be utilized to form part of the AI decision making process. 	"Provide the following details:
1) Describe the target population expected to be covered by the data
2) Explain if the available data suitably represents the defined target population.
3) Explain if the sample data used for training adequately represented/covered the target population.
4) In the event either the sample data and/or target population data coverage was deemed inadequate how was it managed to ensure there were no side effects like bias from such data?"
Data Suitability	6	"2) sufficiently recent and for a long duration, to be meaningful for the algorithm being trained?
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Not Applicable	The data that was provided for the models to be built was based on getting successful incumbent employees to provide assessment input data between Jan and April 2021. The success models were then built based on this data, ensuring sufficient data sample size, reviewed for biases (de-biased), confirmed by SCB, and deployed into production for Early Careers hiring from August 2021 onwards.	"Provide the following details:
1) Explain the considerations in selecting the historical data/snapshot used for training.
2) In the event sufficient historical data/snapshot was not available, how was it managed?
"
Data Suitability	7c	"3) of appropriate quality (accuracy, consistency with comparable sources where feasible)? 
Where such an assessment suggested that DQ issues could impact the reliability of the final model, were adequate steps taken?
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	All data used for models and scoring is automatically screened for outliers, completeness and signs of cheating. We have strict standards for completeness and anomaly detection, and ensure only the most reliable and high quality data is used for model building. Furthermore, when building a model, the statistical properties of all the feature-distributions are verified to ensure the data quality is on par for the machine learning use case. The high quality data is then adjusted for machine learning purposed according standard ML practices. 	"Provide the following details:
1) The steps taken to assess the quality of the data used in the training.
2) The steps taken to manage any DQ issues identified in the previous step."
Data Suitability	7b	"Are the features, or the way in which they are derived, understood by the AI/ ML developers and end users?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	"The assessments are developed in-house, and are understood by the developers. Clients aren’t privy to all the details of how features are derived, as that remains pymetrics Intellectual Property, but they are provided with the factors (which are aggregates of the features) what they mean. 

Pymetrics ourselves designed the assessments and every feature we extract from them. Derived features are used as all features are derived from the assessments, nothing else. The derived features are thoroughly vetted by IO psychologists and are documented in our internal documents and codebase. The pymetrics Technical Manual describes the research and theory of how they were derived, as well as validation information at the aggregate level (per game) but doesn’t comprehensively document every single feature and how it’s computed. The reasons for this are that the derived features are numerous and complex, subject to semi-annual improvement, in addition to this level of information being Pymetrics Intellectual Property. "	"Provide the following details if response is Yes:
1) Are all the features being used clearly understood by the AI/ML developers as well as the users of the system?
2) Are any derived features required and used?
3) How and where are the derived features documented and verified?"
Data Suitability	7a	"Is it possible to confirm the influence of all key features on the outcome hence to justify its inclusion?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	"Data scientists review the inclusion of features for each model we build. 
For each model, there is a list of feature importances that dictates exactly how each feature contributes to the final model score. While this is available to model developers, our product and client facing teams determined that it wasn’t useful or easily interpretable to end users (i.e. clients). So instead, we aggregate features into factors then regress factors onto model scores, which gives a more intuitive understanding of how behaviors impact scores, and this is the model report that’s available on the Client Portal to pymetrics clients"	Explain the inclusion of every feature used as input using its influence or predicting power on the final outcome. This will help remove superflous features in the input and keep them limited to what is required for the algorithm to function.
					
Explainability/ Justifiability	12a	"Is it possible to reliably explain the key factors driving the recommendations, in a way that lay humans can understand at:

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	Yes in lay terms - the assessment recommendations are borne from a mixture of analysis of job content in terms of skills and attributes and combined with the results of current employees who have undertaken the assessment. In essence we have created a success profile based on what good already looks like from the Bank's current employees.	
Explainability/ Justifiability	12a	"1) An aggregate level
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	Pymetrics explains recommendations through the behavioral traits measured/factors. These come directly from the model building explainability layers, and are an aggregate of the feature importances. This aggregate is made with data science and IO methods and has been thoroughly validated and reviewed. The factor importances are reviewed by the consultants and presented to the client for each model, ensuring they are approved by all the stakeholders.	"Provide the following details:
1) Explainability mechanism that is used to support the recommendations made by the algorithm at a global/aggregate level
2) How and where has this been documented and verified?
3) Based on the above, has the list of factors driving the recommendation at an aggregate level, and their relative importance, been validated?"
Explainability/ Justifiability	12a	"2) The level of individual recommendations
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	Pymetrics explains recommendations through the behavioral traits measured/factors. These come directly from the model building explainability layers, and are an aggregate of the feature importances. This aggregate is made with data science and IO methods and has been thoroughly validated and reviewed. For each candidate, a Recruiter report is available to the Hiring Manager/Recruiter. This report outlines which behavioral traits are most important for the model, the candidate's fit against each of these traits, and suggested interview questions based on gaps between the candidate's behavioral traits and those found to be most important for job success. This is an automated report and can be verified through follow up interview with the candidate.	"Provide the following details:
1) Explainability mechanism that is used to support the recommendations at a local/individual recommendation level.
2) How and where has this been documented and verified?
3) Based on the above, has the list of factors driving the recommendation at an aggregate level, and their relative importance, been validated?"
Explainability/ Justifiability	12b	Where reliable explanations are unavailable:			
					
Unjust Bias	10	"Have all potential types of unjust bias been identified, that might be relevant to this use case (i.e., what kinds of outcomes would we want to avoid)? This should be driven by  relevant regulations  - e.g., those around consumer protection - as well as the bank’s own code of conduct/ values as applicable.

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)
"	Yes	All Pymetrics models are proactively de-biased before they are deployed for selection purposes, ensuring no population is advantaged/disadvantaged. In accordance with EEOC regulations, models must pass the 4/5th's rule where performance is compared across demographic groups during the model build. Fairness is assessed by using a withheld set of users for whom demographic information is known, called the bias set. The bias set contains users from each EEOC defined race, ethnicity and gender group. Data scientists always verify these populations are representative and sufficient in sample size. Debiasing on additional demographic groups can also be requested and implemented depending on the geographical relevance and legal requirements for the use case all of which have been considered by the Project team when working with Pymetrics to establish the use case and model. The bias set is tested against each Pymetrics model. The proportional pass rates of the highest passing demographic group are compared to the lowest passing group for each demographic category (gender, race, and ethnicity).This proportion is known as the min bias ratio (also referred to as the Impact Ratio)	"Provide the following details:
1) Are there any direct or indirect drivers/factors (such as gender, religion, geography, nationality etc.) that might favor/disadvantage one section of the target population over another?
2) Have applicable regulations (e.g. GDPR) been identified and used in identifying these factors?
3) How and where is this assessment documented and verified?"
Unjust Bias	8; 9	Has the data set used for training set been confirmed to			
Unjust Bias	8; 9	"1) Exclude any ‘protected variables’ identified under 4a (and any other obvious proxies for the same)
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	"Pymetrics models are debiased for gender and race before being deployed, using EEOC's 4/5ths rule as the minimum debiasing standard for gender and race. For further information on our debiasing approach, please refer to our technical manual. Pymetrics also conducts back testing for gender and race post going live. The cadence of back testing is dependent on the availability of labelled gender and race data being provided by SCB. 

Pymetrics does not proactively debias our models for age differences, but we do test for age differences in our outcome studies for individual clients. To date we have not seen evidence of our selection models disproportionately favoring candidates under age 40. Some client models have favored candidates over age 40, but the most common result is no significant difference between these groups. This has not been tested yet for SCB but can be monitored once the process is live. (Note this will be used in Early Careers hiring only, which targets current and recent graduates specifically and thus older candidates would be the ‘minority’ group in this scenario)."	
Unjust Bias	9	"2) Be representative enough of all relevant categories of data subjects (e.g., gender, race) to provide a fair outcome?
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	See 4.1	"Provide the following details:
Does the training data include sufficient number of samples for all protected groups (for e.g., if gender is a protected category, are there sufficient rows of data representing gender minority groups so that the model does not get biased against them).  Note that minority groups vary across countries - majority in one country could be minority in another"
Unjust Bias	9	"3) Have the recommendations of the model been overlaid with the protected variables to check if bias has crept in despite their exclusion from the training data set?
Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	The goal of the Pymetrics pipeline is to produce a model that is both predictive and unlikely to be biased by gender or ethnicity. To achieve this result, Pymetrics data scientists build hundreds of models for each role, with different combinations of model weights and parameters. The pipeline begins by building a model containing all possible measures from all available games. This model is tuned for best performance before being tested for bias using audit -ai. If the model passes checks for cross validated performance metrics and checks for bias against all demographics in the hold-out bias set, that model is saved for further review. PAGE 82	"Provide the following details:
1) Have the outcomes of the model been mapped to protected variables (if any)?
2) Explain the analysis performed on the outcomes of the model, after mapping the results to protected variables, to identify unfair bias that may have inadvertently crept in and any subsequent actions taken.

This approach helps identify bias that may have been derived from proxy features of the protected variables and may go undetected otherwise"
Unjust Bias	11	Where the above assessment suggests some bias			
Unjust Bias	11	1) is it within regulatory limits if any (e.g., those on Fair lending)? If not, can it be justified based on underlying data and explainability?	Not Applicable	Pymetrics models are monitored for biases as well as changes in applicant populations. In the rare occasion that biased is detected, we work closely with the client to determine the best way to remedy that bias for the specific case.  Changes continue to be monitored to ensure compliance with all regulations.	"Provide the following details:
1) Are there any unfair bias outcomes identified that have been limited/restricted by regulations?
2) Are the outcomes tested and confirmed to be within those limits?
3) In all cases, if bias is not deemed unfair but necessary (e.g., differentiating between Retail and Private Bank customers) - provide justification."
Unjust Bias	11	2) Even if that is the case, is it a position consistent with the bank’s values ?	Yes	Yes SCB have worked alongside Pymetrics to ensure that the setup of the assessments and the assessment itself meets the values of the Bank and it should be noted that following the Pymetrics assessment, successful candidates at this stage also undertake the SHL Value Behaviors assessment. The same is true for the models.	Do any of the outcomes contradict the core values bank stands for (notwithstanding the fact that they are within regulatory limits), and explain the reasoning behind the response
					
Transparency	17a; 17b	Does this use case involve AML / Sanctions screening / Fraud detection?	Not Applicable		If the use case involve AML / Sanctions screening / Fraud detection checklist items 5.2 to 5.4 are not applicable.
Transparency	17a	Has the bank explained its use of AI/ ML for this use case – in a simple, easy to understand manner – to customers, staff and other impacted parties? (e.g., in the T&C, on screen/ face to face/ on the phone when the algorithm is being applied)	Yes	As candidates enter the Bank's recruitment system they are asked to read and accept a Privacy statement that explains how their data is collected and used and includes a section on the use of Automatic Decision making specifically and their right for human intervention and also the deletion of their data. In addition prior to undertaking the assessment, when the candidate enters the Pymetrics system they are asked to read and accept the Pymetrics Privacy statement which fully lays out how their data will be used, the results, how they can request there data and all that is required in terms of the information pertaining to their data	"Provide the following details:
1) What information is communicated to the data subjects with respect to use of AI/ML?
2) How is this information communicated to the data subjects?"
Transparency	17b	On request from data subjects, is the bank able to provide them		Yes, as per the GDPR’s right of access principle.	
Transparency	17b(i); 17b(ii)	1) clear explanations on what data is used to make decisions using AI/ ML algorithms about the data subject and how the data affects the decision	Yes	"Pymetrics Privacy Policy is available from the Pymetrics landing page, and linked here: https://www.Pymetrics.ai/privacy-policy

Other communications that go to candidates in assets like email invitation, welcome video and other elements will be finalized before launch."	"Provide the following details:
1) Are existing channels used to achieve this?
2) If existing channels are not adequate, how have they been enhanced to achieve this?
3) What information / explanation on the data used for the AI/ML process will be provided and how will they be presented to the data subjects?
"
Transparency	17b(iii)	2) clear explanations on the consequences that AI/ ML driven decisions may have on them	Yes	"Pymetrics Privacy Policy is available from the Pymetrics landing page, and linked here: https://www.Pymetrics.ai/privacy-policy. A copy of SCB's Recruitment Policy is also attached

Other communications that go to candidates in assets like email invitation, welcome video and other elements will be finalized before launch."	"Provide the following details:
1) Are existing channels used to achieve this?
2) If existing channels are not adequate, how have they been enhanced to achieve this?
3) What information / explanation will be provided on the impact of using AI/ML on the subjects, and how will the information be presented to the data subjects?
"
Transparency	17c	Does the bank provide Data subjects 			
Transparency	17c(i)	1) with channels to enquire about, submit appeals for and request reviews of AI/ML-driven decisions that affect them	Yes	All candidates will have the right to have their results of the pymetrics’ games reviewed and are made aware of this right (and who to contact) in an FAQs document made available to them on the careers website	"Provide the following details:
1) Are existing enquiry/appeal channels used to achieve this?
2) If these channels are not adequate, how have they been enhanced to achieve this?"
Transparency	17c(ii)	2) the ability to provide verified and relevant supplementary data (e.g., through online data management tools such as privacy or personal data dashboards) which will then be taken into account by the Group when performing a review of the original decision	Yes	"See above. Candidates will be provided a contact whom they can discuss their request with and share information as necessary; there are relevant escalation points where further help is needed to handle the case, e.g. the assessment team / SMEs at the vendor. Note that the purpose of using pymetrics in the selection process is to remove any biases, thus the data supporting any request for a review would be considered appropriately. E.g. should a candidate be requesting a review following a rejection from the process, it would be inappropriate for us to seek any additional data (such as a resume) which would not also be considered for other candidates as part of the selection process.   
"	"Provide the following details:
1) Are existing review channels used to achieve this?
2) If these channels are not adequate, how have they been enhanced to achieve this?"
					
Stability/ over-fitting	13; 14	"Has the model been assessed for predictive power (accuracy), stability and over-fitting?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	All models are screened for accuracy, recall, precision and other relevant metrics. The stability of these metrics is established with k-fold cross-validation as well as validation on hold out sets to ensure there is no overfitting. Depending on the use case of the model, we optimize for overall accuracy, recall or precision. As false negatives are nearly always less desirable in hiring, the models are typically selected to maximize recall. Regardless of the specific use-case, all models must meet the internal standards of > 65% accuracy and > 70% recall.	"Provide the following details:
1) Explain the metrics used to measure model accuracy and the results.
2) Tests performed for stability and over-fitting of the model."
					
Assessment/Validation	15a	Has the AI/ ML algorithm been assessed:			
Assessment/Validation	Sec 4.1	1) By Group Model Validation (GMV) team to come under the definition of a Model under the Model Risk Policy/ Standards? 	No	"Reviewed as Non Model  based on the reviewer's comments -
Output from the tool is a recommendation as to how best a candidate fits into the role that he/she was assessed for. It is understood that the output produced by the tool provides information as part of a business process (hiring) without directly impacting the process, in which case the tool is considered as an analytic tool. Analytic tools are not considered a model under the Group Model Risk policy and standards."	"Provide the following details:
1) The assessment needs to be signed off by the MRM team
2) Where is the assessment available?"
Assessment/Validation	15a	2) If the answer to 7.1.1 is "Yes" has the model been through appropriate assessment/validation from the MRM team?	Yes		"Provide the following details:
1) Confirm if the model has been tested and signed off by the MRM team.
2) Where are the test results documented?"
Assessment/Validation	15a	3) If the answer to 7.1.1 is "No" or "NA", has the model been peer-reviewed by a suitable/independent SME other than the one developing/procuring it?	Not Applicable	Every model is developed by a data scientist SME, AND subjected to an automated validation checklist, AND peer-reviewed by a second data scientist SME. This is documented on an internal GitHub repo where the notebook used to create each model is stored.	"Provide the following details:
1) Confirm if the model has been tested and signed off by an independent SME?
2) Where are the test results documented?"
					
Monitoring 	16	"Has an assessment of the required levels of monitoring during and after go-live taken place?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	"During go live In addition to UAT and regression testing, all models undergo three stages of human review before deployment (builder, a manual checklist, and review by a data scientist). Outputs are monitored by data analysts and client support staff for candidate flow, potential bias, and other user-flow statistics. If any anomalies are detected, they are raised to the data analytics team, who investigates further.
As a BAU process , Pymetrics will be monitored periodically. This will measure the extent of Efficiency, Effectiveness, Diversity and Candidate Experience."	"Provide the following details:
1) Explain the metrics that will be monitored on an ongoing basis
2) Periodicity/Frequency of such monitoring.
3) Agreed process for monitoring."
Monitoring 	18a	Has the agreed level of monitoring been put in place?	Yes	Periodic reviews are carried out annually before each campaign to make sure no band adjustments are required.	"Y = Periodic reviews are in place/ planned
N = No Periodic reviews are in place/ planned
NA = Not Applicable
"
Monitoring 	18b	Are the results from such monitoring accessible and actionable (i.e. can the people looking at it actually do something with it, or is it too overwhelming and complex)	Yes	Pymetrics will share the results with the Bank through regular presentations to the client stakeholders, in a documented form. These will be discussed and changes made where necessary (e.g. if thresholds are breached) and could result e.g. in the model being tweaked or refreshed.	"Provide the following details:
1) How and where will the results of the monitoring be documented/ accessed?
2) Agreed process of reviewing/ managing the monitoring results, especially when exceptions are observed or thresholds are breached"
Monitoring 	18b	Have BAU processes and/ or controls been put in place to ensure appropriate corrective action and feedback in a timely manner?	Yes	The Pymetrics project team are working through the processes and controls which will be documented as part of the project. For candidates this will involve an FAQ section held on SCB's Early Careers website.	"Confirm if the monitoring process as explained in 8.1 to 8.3 above, along with necessary controls have been operationalised/ implemented in BAU.
"
Monitoring 	18b	Is there adequate monitoring in place to ensure post go-live that the data being fed to the AI/ ML model meets the minimum quality and representativeness expectations established during development/ testing	Yes	All models are monitored post go-live to ensure models are performing as expected. Pass rates of all models are monitored so that the expected volume of users passed by the model meets expectations (accounting for data drift). Voluntary demographic data provided by users are used to test for adverse impact at set intervals after the models has gone live (accounting for model bias). Results are reviewed by automated systems and human overseers (such as the client success manager) through internal dashboards. Data drift can be calibrated through adjustments of the passing score (aka the "bands"), providing changes to bands do not lead to bias. Model bias can be calibrated through changing bands or through tuning and redeploying the model.	"Provide the following details:
1) The data quality checks being performed, including for e.g., data drift 
2) The process of monitoring the results of checks
3) The process of managing the exceptions identified."
					
Auditability 	21	"Are appropriate snap-shots of the AI/ ML algorithm and supporting data are being retained throughout the lifecycle (and beyond if needed, depending on applicable data retention regulations in the country)?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	Yes, all models are deployed with backup data sufficient to rebuild the model without containing PII of any individuals used to build the model. User-level data is stored in a separate location and is stored for at least the minimum time required by the region in which the individual is protected, or until the individual requests that their data be removed from the Pymetrics platform. As model training data is abstracted from user-level data, models can still be reconstructed even if user-level data has been removed from the system. Supporting data (or "snapshots") are available through a sanitized data warehouse available to members of the Pymetrics data science team and select members of the engineering team. Supporting data are retained in perpetuity. 	"Provide the following details:
1) Have all snapshots been retained, were there any exclusions?
2) Where and how is the information on snapshots retained and accessed?
3) Who has access to the information on snapshots?
4) How long will the information on snapshots be retained?"
Auditability 	25	"Are human decisions in developing and tuning the algorithm documented along the way

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	Yes the Bank Project team which holds specialists qualified in the use of assessments tools have worked closely alongside Pymetrics to select incumbents across the multiple locations that the assessment will be deployed in to undertake the assessment providing the data for evaluation in order to define the scoring parameters for the SCB assessment and also video interviewing. It should be noted that Pymetrics also manually (human intervention) check that the results meet fairness requirements in order to avoid adverse impact. All decisions made by the data scientists while tuning the model to prioritize the client's needs are logged and retained in perpetuity.	"Provide the following details:
1) Have all human decisions (during development/tuning) been retained, were there any exclusions?
2) Where and how is the information on human decisions retained and accessed?
3) Who has access to the information on human decisions?
4) How long will the information on human decisions be retained?"
Auditability 	26	"Are any automated decisions (e.g., those made on an Auto-ML platform) relating to parameters/ tuning available for review at a later stage?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	"Yes, all automated decisions are reviewed before deployment. Pymetrics adheres to a strict human-in-the-loop process, by which all models are reviewed in the following method:
- An overseeing data scientist builds the model. All automated and human decisions are checked and recorded
- A second data scientist reviews the model and checks all assumptions
The model is reviewed by internal client advocates (such as client partners, client success managers and/or industrial-organizational psychologists)
All automated decisions are redundantly stored in the build logs, in the model review template, and the model delivery ticket given to the internal client advocates. All Pymetrics staff have the ability to view high-level decisions about models, and the data science team has visibility to the full build logs. This information will be stored in perpetuity. "	"Provide the following details:
1) Have all automated decisions (related to parameters during development/tuning) been retained, were there any exclusions?
2) Where and how is the information on automated decisions retained and accessed?
3) Who has access to the information on automated decisions?
4) How long will the information on automated decisions be retained?"
Auditability 	21	"Are mechanisms in place to record 
(a) any human decisions to reject recommendations made by the AI/ ML algorithm and 
(b) any corrective action taken to the way in which the model works as a result?

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (3P)"	Yes	For Parts 1 - 3 - the goal of using debiased models is to steer clear from requirements for humans to override decisions as this in itself would introduce potential bias/personal subjectivity. This is against pymetrics recommended use of the product. For part 4, Pymetrics conducts periodic model refreshes which is essentially a rebuild of the model to incorporate new successful employees and potentially review and remove those employees that no longer meet performance standards required for model inclusion. Pymetrics does not make any changes to the model outside of this controlled model refresh process. Therefore Overrides performed by recruiters or hiring managers will not influence the success model used. 	"Provide the following details:
1) Once operationalised, where humans override recommendations / decisions made by AI/ML engine, how are these overrides recorded?
2) Where and how is the information on human overrides retained and accessed?
3) Are all overrides by a human subsequently assessed/analysed, and are there any exclusions to this analysis?
4) Are any corrective actions on the model, based on the overrides performed on the model output, recorded?  Describe where and how?"
					
Enablers	23(i)	"Do those building/ testing/ validating/ operating the use case have appropriate expertise? (including third parties)

Requires 3rd party/ vendor inputs if the AI is sourced from a 3rd party (In house/ 3P)"	Yes	Yes as, as laid out in the Justifiability section, Pymetrics have a 140 page technical document laying out the operating model behind the assessments, and how they are validated. They also have a team of Data Scientists working on the creation of the assessment and models for each individual company and all and any assessments go through stringent checking processes before being signed off which also requires the Client, in this instance the Bank's sign off.	"Provide the following details:
1) Expertise/experience levels of the individuals/teams involved in the lifecycle.
2) If it involves external/third parties - how have their expertise been assessed."
Enablers	Tech Council Process (e.g. PED)	Has the relevant technology council approved the technology platform used to develop and deploy the AI/ ML algorithm?*	Yes	As well as putting the project fully through the relevant sign off committee's, the Project team is working alongside the AI team and also the provider to ensure that the expertise is at the required level. This project has fully been through all CGC sign offs and independently signed off by the Singapore & HK technical teams alongside having passed through both the Singapore OWG and OC process	"Provide the following details:
1) What tools and technology platforms are used in the development/deployment?"
					
Others	Sec 4.3	Have incremental security risks (e.g., risk of poisoning of training data, IP theft) on account of the Ai/ML use been assessed?	Yes	Going through the SIA is part of all projects being submitted and this is in progress and due to be signed off on the 5th July for Pymetrics. Previous answers contain further details of the security methodology that is put in place by metrics to ensure that the system/assessment is secure. All security and compliance governance is up to date. Quarterly Vendor Service Management reviews are carried out and anything outstanding is highlighted and discussed.	"1) Complete the Security Assessment which is available here. The assessment needs to be signed off by ICS
2) Where is the assessment available?"
Others	Sec 4.3	Has the broader security impact assessment been completed?*	Yes	Going through the SIA is part of all projects being submitted and this is in progress and due to be fully signed off on the 16th July for Pymetrics.	
Others	Sec 4.4	Have all operational resilience considerations been taken into account ahead of go live?	Yes	The Pymetrics assessment has passed through all of the relevant resilience assessments as relevant to the Banks on boarding of technology requirements	
Others	Sec 4.5	Where a 3rd party vendor is involved in building/ testing/ deploying the model, or in providing the data used in the model, have all appropriate third party risk management considerations been taken into account?	Yes	Yes, this project has been through all necessary assessments inclusive of the TPSA	   draft few questions based on RAI principles based on the approach