Learner Responsive National Success Rates Tables for the academic year 2008/09

Guidance Notes

What is new in the 2008/09 Tables?

- The benchmarking data for 2008/09 have been renamed ‘National Success Rates Tables’.
- Two measures that are no longer used by the inspectorate have been removed – Retention Rate (including continuers); Achievement Rate (known outcomes). All other measures are retained.
- Success Rates by Local Authority are shown for the first time. This is done in two ways: by the Local Authority of the learner’s home address; and by the Local Authority of the provider’s delivery location (for the year 2008/09 only).

Success, Retention and Achievement Rates in Further Education Providers in England The National Success Rates Tables for Learner Responsive provision (previously known as ‘Benchmarking Data’) set out levels of success, retention and achievement in further education (FE) providers in England for the period 2006/07 to 2008/09. The Data Service publishes National Success Rates Tables annually. This release of Learner Responsive National Success Rates Tables replaces the Benchmarking Data 2005/06 to 2007/08 published in Summer 2009. It is consistent with the restated Learner Responsive Qualification Success Rates reports for 2008/09 published in July 2010 and expands upon the information made available in the June 2010 Statistical First Release (DS/SFR7).

What is the purpose of the National Success Rates Tables? The publication of National Success Rates Tables supports providers in raising the standard of their work. The Tables allow providers to assess their performance, and assists the planning of action programmes to improve the success, retention and achievement rates of their learners.

The term National Success Rates Tables, rather than ‘Benchmarking Data’, is used throughout this guidance. National Success Rates Tables is used to denote a context and reference point for comparison, not a standard of best practice.

As part of Framework for Excellence, all providers agree annual targets for improving learner success rates with their funding bodies in their development plans. Target setting informed by National Success Rates data is an integral part of providers’ strategies to continuously improve. What do the tables show? The Data Service approach to publishing National Success Rates Tables is to publish a manageable amount of information, drawing on existing statistical measures.

The National Success Rates Tables data have been derived from providers' Individualised Learner Record (ILR) returns and furnish a range of national and provider statistics for success, retention and achievement.

This year four categories of National Success Rates Tables Data are presented:

- overall data tables showing national averages of success, retention and achievement rates, plus spread of variability, by college type, notional level, age, gender, disability, ethnicity, length and locality.
- national data for success rates, retention and achievement by sector subject area.
- qualification data showing success, retention and achievement by notional level and qualification type
- institution tables showing success, retention and achievement by provider, qualification type, age, gender, disability, ethnicity and subject area

The underlying methodology used to generate the data changed to the 'Demand-Led Funding' model in 2008/09. A different methodological model had been used for 2007/08 and prior years. The National Success Rates Tables draw data from both these sources: national success rates for 2006/07 and 2007/08 are calculated according to the methodology in force in 2007/08; national success rates for 2008/09 are calculated according to the Demand-Led Funding methodology.

What information do the Tables contain? There is one retention rate and one achievement rate shown in this publication. This has changed from last year's publication. The definitions of the success, retention and achievement rates can be found in the 'How are the Tables calculated' section below.

Short qualifications, where the student expects to complete in fewer than 24 weeks, are distinguished from longer qualifications. The tables also show national success rates for short qualifications of less than five weeks expected duration and short qualifications of between 5 and 24 weeks expected duration.

The measures Retention Rate (including continuers) and Achievement Rate (known outcomes) have been removed. These are no longer used as indicative measures by the inspectorate bodies to assist them in reaching judgements on individual providers' results in comparison with national figures.

Exclusions Key Skills qualifications have not been included in the National Success Rates Tables owing to inconsistencies in the way that they were recorded on the ILR. UFI, Functional Skills and OLASS qualifications are also not included. This is consistent with the approach taken in Learner Responsive Qualification Success Rates reporting. How are the Tables used?

- They allow providers to assess their performance and assist their planning of action to improve the success, retention and achievement rates of their learners.
- They support providers in the process of setting targets as part of their development plans set out by Framework for Excellence.
- They inform the process of performance review by the Skills Funding Agency and by Local Authorities.

Who are the Tables used by? Providers can contextualise their performance by comparing their results with the published National Success Rates Tables. This information will support the process of setting targets for 2010/11 and beyond. Individual provider results will also be used by the Skills Funding Agency and by Local Authorities to compare and evaluate the performance of providers.

Where a provider’s performance differs from the National Success Rates Tables data, the provider may wish to explore the reasons for this by calculating success, retention and achievement rates for particular parts of its provision and comparing these with statistics for similar provision at national level. To facilitate this comparison, a more detailed breakdown of the National Success Rates Tables data by type of qualification, and individual qualification aim is also available.

In some cases there may be a difference between provider statistics and the National Success Rates Tables data because the mix of qualifications at the provider differs significantly from the national mix. This means that for comparison purposes the National Success Rates Tables data should be adjusted. An example of how to adjust the data is shown in the definitions section.

How do I view the Tables? The data for 2008/09 can be accessed at the National Success Rates Tables pages of the Data Service website. The tables are presented as a set of spreadsheets that can be downloaded by users for further analysis.

HOW ARE THE TABLES CALCULATED? Summary of calculation method The method takes data for each institution from the following ILR returns to create the National Success Rates Tables:

- F05 ILR (2005; 2004/05)
- F05 ILR (2006; 2005/06)
- F05 ILR (2007; 2006/07)
- F05 ILR (2008; 2007/08)
- F05 ILR (2009; 2008/09)

F05 2004/05 is used to provide information on learners starting qualifications that were expected to end in 2006/07 or later, i.e. courses of three years or more duration. For example, learners on a three-year GNVQ programme starting in 2004/05 and expecting to end in 2006/07.

Learners and their qualifications are matched across the five years of ILR returns using the learner reference, course code, start date and expected end date to calculate the number of starters at the beginning of each programme, retention across the whole programme, and success and achievement levels. Only qualifications that learners expected to complete between 2006/07 and 2008/09 are included in the publication.

The National Success Rates Tables data is built from cohort level, a cohort being learners studying a particular qualification over the same duration and expecting to end in the same teaching year.

**Presentation issues**

- Where the overall number of starters for an aim nationally is less than 30 at fewer than 2 institutions, results and percentiles are not shown.
- The percentage breakdown of number started may not add up to 100% due to rounding.
- The institution data does not show success rates for qualifications with fewer than five achievers. This prevents the possible identification of known individuals.
- Since the data is calculated at qualification level, learners studying more than one qualification will appear once for each qualification.

**A, AS, A2 and GCSE grades** The variable quality of data in the 'grade' field of the ILR means it is not yet possible to accurately differentiate the ranges of GCSE grades. In the National Success Rates Tables the number of GCSEs achieved is calculated as the number of GCSEs achieved at grades A\* to G for all colleges.

Qualification Level National Success Rates Tables show the percentage achieving a 'high grade' calculated from the grade field in the ILR to provide an indication of the A\* to C achievement rate for GCSEs, and the A to B achievement rate for 'A' levels.

**The Effect of Widely Differing Mixes of Qualification Type** The example below shows how the National Success Rates Tables can be adjusted to match the mix of qualifications at an individual college, where the mix of qualifications at the college is significantly different from the national mix.

The example relates to adult learners studying at Notional NVQ Level 3 in a general further education college. The starting point is the average retention and achievement rates for different level 3 qualifications for all general further education colleges and for our example college.

| Table 1. Notional level 3 qualifications, learners aged 19 and over | |---------------------------------------------------------------| | **Average for GFEC / TC** | **Example College** | | Retention Rate % | Achievement Rate % | Breakdown of enrolments % | Breakdown of enrolments % | | All Qualifications | 78 | 72 | - | - | | GCE A/AS/A2 | 67 | 64 | 18 | 10 | | GNVQ | 68 | 77 | 9 | 5 | | NVQ | 82 | 73 | 17 | 50 | | Other long | 83 | 64 | 56 | 35 | In our example college, the level 3 provision for adults includes a higher proportion of NVQs, and a lower proportion of GCE A/AS/A2 levels, GNVQs and Other qualifications than the national mix. In order to check whether the published National Success Rates Tables data for all level 3 qualifications is an appropriate comparator, the following calculation may be used:

Average GFEC/TC retention rate = 78% GFEC/TC retention rate for mix of qualifications in example college comprising: 10% GCE A/AS/A2 retention rate of 67% = 6.7 5% GNVQ retention rate of 68% = 3.4 50% NVQ retention rate of 82% = 41 35% Other long retention rate of 83% = 29.05 80.15 rounds to 80%

The same approach can be used for achievement rates:

Average GFEC/TC achievement rate = 72% GFEC/TC achievement rate for mix of qualifications in example college comprising: 10% GCE A/AS/A2 achievement rate of 64% = 6.4 5% GNVQ achievement rate of 77% = 3.85 50% NVQ achievement rate of 67% = 33.5 35% Other long achievement rate of 75% = 26.25 70.00 rounds to 70%

As can be seen, the adjusted National Success Rates Tables data are similar to the original National Success Rates Tables data, despite the very different mix of qualifications in the example college. In practice, mix of qualification types is unlikely to be a significant factor for most colleges. This method can be used on success rates and on achievement and retention rates.

Age group A learner’s age group is calculated from their age as at 31 August in the year they started their qualification. Learners of unknown age are included in the age group 19 and over. Learners under 16 years are included in the 16-18 age group. All tables show the National Success Rates Tables data divided into three age groups: ‘16-18’, ‘19 and over’, and ‘All Ages’.

Short qualifications A qualification is ‘short’ if it has an expected length of less than 24 weeks. In practice the majority of short qualifications are of 12 weeks duration or less. Short qualifications (excluding short Key Skills qualifications) are shown separately, since the success, retention and achievement rates for these qualifications are significantly different from those of longer qualifications. In addition, the tables show National Success Rates Tables data for short qualifications split into two subsets: less than five weeks expected duration and between 5 and 24 weeks duration.

Notional NVQ level Qualifications are grouped according to their NVQ level or notional equivalent according to the categorisation of each qualification on the Data Service’s qualification database. The levels are: level 1 includes qualifications at level 1 and level ‘E’ (entry level), such as NVQs, foundation GNVQs and other foundation or pre-foundation qualifications level 2 includes level 2 NVQs, intermediate GNVQs and precursors (BTEC first certificate or first diploma, City and Guilds Diploma of Vocational Education at intermediate level), GCSEs and other intermediate level qualifications

level 3 includes level 3 NVQs, advanced GNVQs and precursors (BTEC national certificate or national diploma, City and Guilds Diploma of Vocational Education at national level), advanced VCEs, GCE A, A2 and AS levels and other advanced level qualifications

level H all level 4 and 5 qualifications including HNCs, HNDs, access to HE qualifications, NVQs at levels 4 and 5, and other higher level professional qualifications.

Level X Qualifications with unknown or unspecified level

Qualification Codes The qualification codes shown in the qualification level data tables can be mapped to the qualification codes used in ILR F05 (from the learner aims database (LAD)) using the hierarchy table that can be found at: http://providers.lsc.gov.uk/LAD/downloads/MiscDownloads.asp (see table labelled Mapcodes).

In this hierarchy table the Mapcode column corresponds with the qualification code shown in the qualification data tables of the National Success Rates Tables.

Number of starters The 'number of starters' is the number of enrolments on qualifications where the learner was expecting to complete the qualification that college year. Details to note on the definition include: (a) the number started excludes any learner who transferred onto another qualification at the same institution. The qualification the learner transfers into will be included as a start on the new qualification. (b) learners who start on a qualification expected to last at least 168 days and withdraw within 42 days of starting, qualify for an SLN payment yet have received no payment, are considered as not funded and are excluded from the count of starters. (c) each qualification a learner is enrolled on is shown as a separate 'start' (d) the 'number of starters' for earlier years includes some provision not funded by the then LSC. (e) A learner on a two year programme who began their studies in October 2007 would appear in the results for 2008/09 as this is the college year in which they expected to complete their qualification, even if they withdrew before August 2008.

Success rates Success rates are calculated as the number of qualifications achieved divided by the number of starters who do not transfer out (i.e. to another institution). It can also be derived by multiplying achievement rate by retention rate.

Retention rate The retention rate is the number of learners completed, divided by the number of learners who started the qualification, excluding transfers out. For programmes of study of two years or more, retention is calculated across the whole programme, i.e. from the start to the end of the qualification.

Achievement rate The achievement rate is the number of qualifications learners have fully achieved divided by the number of completed qualifications. This denominator includes those completers recorded with unknown outcomes in the ILR, for example outcome code 4 'exam taken but result not known' or outcome code 5 'learning activities are complete but exam has not yet been taken'. Partial achievements are not included as achievements.

**Measures of provider variability**

Measures of provider variability (percentiles) for success, retention and achievement rates enable colleges to compare their results against the range for the sector or particular groups of colleges.

The results in this publication show the rates which:

- 10% of providers meet or surpass (90th percentile)
- 25% of providers meet or surpass (75th percentile)
- half of providers meet or surpass (median or 50th percentile)
- 75% of providers meet or surpass (25th percentile)
- 90% of providers meet or surpass (10th percentile).

Measures of variability are published where there are sufficient numbers of providers to calculate a meaningful result. The measures of variability are calculated at provider level in order to provide information on variation between providers. This is different from the mean success, retention and achievement rates, which are calculated as the average rate for all the relevant enrolments, weighting each enrolment equally.

The measures of variability weight each provider equally. This means that learners in smaller providers have a greater bearing on results than those from larger providers. The results for small groupings, such as qualifications being studied by adults in sixth forms, will be affected by this weighting more than others.

In the case of the Local Authority post-code based percentiles, these simply measure the variability of the subset of providers with locations in that authority (A23) or learners resident in that authority (L17). In the latter case particularly, the equal weighting of each provider may have a significant bearing on the results for authorities where many small cohorts of learners travel outside their home region to follow a course of study.

The differences between the two methods can be seen by comparing the average or 'mean' rate, with 50th percentile or 'median'. In many cases the difference is slight, whereas in others such as notional level 1 qualifications in sixth forms, the difference is greater. In this case the median is higher than the mean due to a number of providers with relatively few learners having high levels of retention and achievement. These providers are given the same weight in the calculation of the median as other sixth forms with a larger number of learners thereby increasing the median provider result.

Both the mean success, retention and achievement rate and the measures of variability are valid and useful measures, depending on whether the overall performance of the sector is of interest (mean success, retention and achievement rates) or the variability between providers is the focus (measures of variability).

**Example Code**

Example code describing the standard algorithms used to calculate success, retention and achievement rates for the Learner Responsive Qualification Success Rates reports and the National Success Rates Tables data 2008/09 for further education providers in England are available on the Data Service website via the Technical Definitions pages. How do I find out more? If you would like any more information about National Success Rates Tables please contact our Service Desk on 0870 267 0001 or email servicedesk@thedataservice.org.uk. Please be aware that the Data Service is responsible for producing NSRT reports in accordance with the Learner Responsive Qualifications Success Rates methodology approved by the Young People’s Learning Agency & the Skills Funding Agency. The methodology is produced by the Strategic Analysis Group of the Young People’s Learning Agency following the principles set out by the Data Harmonisation Group. See: http://www.theia.org.uk/harmonisingsuccessrates/dataharmonisationgroup/ The methodology is available at the National Success Rates Tables pages of the Data Service website.

Providers will find answers to most of their queries regarding National Success rates Tables in the guidance document provided or in the frequently asked questions at the National Success Rates Tables page on the Data Service website. It is strongly recommended that providers refer to these sources before contacting the Data Service.

If providers need further help, the Data Service will be pleased to receive queries regarding National Success Rates tables. However, when logging a query with the Data Service please provide a detailed explanation of the query being raised. Please also note that it may take longer to resolve queries in this way as they may need to be reviewed by different members of the Harmonisation Group.