Building trust in assessments: The research behind DataCamp’s Data Science Certification
By: Pieter Moors
Tags: Certification, Validity, Reliability, Psychometrics, Data Science

Zooming in on the reliability and validity of skill assessment in our Data Science Certification

This is the first blog post focused on research topics from the Data Science team at DataCamp. We hope to add to our Engineering Blog with posts about psychometrics, statistics, predictive models, and applied data science. Stay tuned for more!

Certifications must be trustworthy and credible to be valuable. In this blog post, we evaluate the quality of the assessments involved in DataCamp’s Data Scientist certification with respect to their reliability and validity. These metrics are critical for understanding how the assessments are functioning and add evidence to the value of the certification.

DataCamp’s Data Science Certification

In March 2021, DataCamp launched certification for entry-level Data Scientists. The goal of this certification is for people to signal that they have the skills required to work in a data science role. These skills include programming, data wrangling, and data analysis, writing a report, and effectively communicating a case study project to a non-technical audience.

The Data Science certification consists of three steps. First, a user has to pass six timed skill assessments. These skill assessments aim to measure skills such as coding for production, data management, exploratory analysis, statistical experimentation, and model development (see Table 1). Second, a coding challenge needs to be completed where a user tackles a real-world data science problem. Third, a user has to complete a case study consisting of a technical report and an oral presentation on that report. When a user passes all three steps, they are awarded the Data Science certificate and are provided with access to DataCamp Career Services for resume review and career consultation.

A critical hurdle in acquiring the Data Science certification is to pass six timed assessments. Skill assessments were introduced to DataCamp in 2019 and they provide an easy and efficient way for users to assess how their current data science skills stack up. 14 different assessments are currently available on DataCamp, 11 of which are used for the Data Science certification. Table 1 lists the different assessments a user has to pass for the Data Science certification, along with the skills they aim to measure, technologies in which they can be taken, and target scores needed to pass the test.

Table 1. Overview of skill assessments for Data Science certification.

Each assessment consists of a computerized adaptive test that aims to efficiently estimate a user’s skill level by presenting items that are tailored to the skill level estimated during the assessment. An assessment consists of 15 items and takes about 10 to 15 minutes to complete. In the end, a score between 0 and 200 is communicated to the user. The scores have been calibrated such that they follow a normal distribution with a mean of 100 and a standard deviation of 30. Thus, as can be judged from Table 1, the passing scores are on the high end of the scale, indicating that a user needs to be highly skilled on each of the dimensions assessed in order to get certified.

The trustworthiness and credibility of the assessment scores are critically important. In this blog post, we evaluate the quality of the assessments included in the Data Scientist certification with respect to their reliability and face validity.

What determines an assessment’s quality?

For any assessment, reliability is an important quality metric. First, a high-quality assessment should be internally consistent. That is, exposing different items to the same user should lead to the same test result. Second, it should also consistently yield similar scores when a user repeats the test, even when that test consists of entirely different items — which is highly likely in a computerized adaptive test. Reliability can be quantified using a host of different metrics, and in this blog post, we focus on two: split-half reliability to evaluate internal consistency and test-retest reliability to evaluate the consistency across measurement occasions.

Whilst reliability measures indicate the scoring consistency within an assessment or across assessments, validity refers to the measures having meaning outside of the assessment context. One such validity measure is face validity which refers to whether a test covers the skills it is said to measure. Here we evaluate face validity by quantifying the extent to which users agree with the test scores communicated to them.

Data set

Our data set consists of all assessments that counted towards the Data Science certification and were completed between March 1, 2021 and May 31, 2021. This 3-month window allows us to evaluate the stability of all reliability metrics since certification was launched. We only consider assessments where a score was reported, and no unusual test behavior was detected (e.g., the test was completed too quickly).

Split-half reliability

Split-half reliability involves randomly splitting each assessment in half (i.e., one half contains 8 items and the other 7), computing test scores based on each test half and correlating these scores across users. If different sets of items lead to the same score for a user, split-half reliability will be high. Table 2 summarizes the split-half reliabilities of each certification assessment, along with the 95% bootstrapped confidence interval. Reliability coefficients are expressed as Pearson product-moment correlations and therefore range between -1 and 1. Reliability values in the range .8 — .9 are generally considered to be very good for standardized tests. Given that computerized adaptive tests such as ours show different items to different users, it can be harder to achieve reliability that high. This is not the case however as most assessments show reliability above .8 indicating very high internal consistency.

Table 2. Split-half reliability for all assessments included in the certification.

Figure 1 summarises the stability of split-half reliability over time. Here, the 28-day rolling average is depicted for each assessment. We monitor these metrics continuously in order to inform the development of the product, and this view provides those insights. For most, split-half reliability remains practically constant across the three-month period and it does not differ significantly between technologies. That is, an assessment taken in either R or Python is equally reliable.

Figure 1. Split-half reliability (28-day rolling) for all certification assessments. The shaded areas indicate the bootstrapped 95% confidence interval.

Test-retest reliability

Test-retest reliability was computed by taking all test scores from users who had completed the same assessment within 7 days from each other and correlating these test scores across users. In Table 3 test-retest reliability for the full three-month period is depicted, along with the bootstrapped 95% confidence intervals. In general, test-retest reliability is slightly lower compared to split-half reliability. This is to be expected as there can be a delay up to 7 days before retaking. Furthermore, at the end of an assessment, relevant remediation material is recommended to the user which can lead to interim learning, consequently causing score increases. Test-retest reliability assumes that the user’s true score has been stable between measurement occasions, which is not an assumption that holds in our case. Even still, most values are in the .7 — .8 range indicating good test-retest reliability, especially considering interim learning.

Table 3. Test-retest reliability for all assessments included in the certification

Figure 2 summarises the evolution of the 28-day rolling average of test-retest reliability across the three-month period. Compared to split-half reliability, test-retest reliability is much more varying, and the variability differs strongly between assessments. Most of this variation can be explained by sizable differences in the volume of retakes within our predefined window. For example, a conspicuous data point is the current test-retest reliability of Machine Learning Fundamentals with R. This assessment has among the lowest number of retakers of all certification assessments, and, as visible in the Figure, has the largest standard error and the largest volatility of the reliability estimate. A deeper analysis revealed however that the current value suffers from a restricted range. That is, most retakers in the last month had above-average scores which restricted the range of the observed scores and consequently attenuated the correlation.

By monitoring these metrics continuously, we can proactively investigate issues, and help ensure the assessment quality remains high.

Figure 2. Test-retest reliability (28-day rolling) for all certification assessments. The shaded areas indicate the bootstrapped 95% confidence interval.

Face validity

When a user completes an assessment, they view a summary page that communicates their score in the 0–200 range and places that score in the population distribution of scores. Together with this feedback, users are asked whether they agree with the reported score (yes/no). At DataCamp, we use this measure as an indication of face validity. If scores are in line with how users judge their own skills, the assessments are accurate with respect to how users perceive their own skills. In Table 4, the percentage of users agreeing with their score is depicted. Averaging at around 85–90%, it is clear that most users agree with the skill level the assessment produced.

Table 4. Face validity for all certification assessments.

Figure 3 depicts this metric as a rolling average, making it clear that this metric is quite stable across time. In sum, all assessments have high face validity.

Figure 3. Percentage agreement with score (28-day rolling average) for all certification assessments.

Conclusion

A certification is only valuable if it is trustworthy and credible. In our Data Science certification, timed skill assessments are an important first step towards becoming certified. We thus continuously strive to keep the measurement quality of our assessment to a high standard. In this blog post, we evaluated the measurement quality of all skill assessments included in DataCamp’s Data Science certification through their reliability and face validity. All analyses indicated that all skill assessments show good split-half and test-retest reliability as well as good face validity. In sum, this provides evidence that the skill assessments in the Data Science certification provide high-quality measurement. As an important step in the certification process, this evidence helps build trust and credibility in the Data Science certification.

Interested in topics like this? Join us, we are hiring across many departments including Data Team roles.