Denosumab for the prevention of osteoporotic fractures in postmenopausal women

Evidence-based recommendations on denosumab (Prolia) for preventing osteoporotic fragility fractures in postmenopausal women.

# Guidance
Denosumab is recommended as a treatment option for the primary prevention of osteoporotic fragility fractures only in postmenopausal women at increased risk of fractures:
who are unable to comply with the special instructions for administering alendronate and either risedronate or etidronate, or have an intolerance of, or a contraindication to, those treatments and
who have a combination of T-score, age and number of independent clinical risk factors for fracture (see section 1.3) as indicated in the following table.
T-scores (SD) at (or below) which denosumab is recommended when alendronate and either risedronate or etidronate are unsuitable
Number of independent clinical risk factors for fracture
Age (years)
-r older
  Treatment with denosumab is not recommended.
Denosumab is recommended as a treatment option for the secondary prevention of osteoporotic fragility fractures only in postmenopausal women at increased risk of fractures who are unable to comply with the special instructions for administering alendronate and either risedronate or etidronate, or have an intolerance of, or a contraindication to, those treatments.
For the purposes of this guidance, independent clinical risk factors for fracture are parental history of hip fracture, alcohol intake of 4 or more units per day, and rheumatoid arthritis.
People currently receiving denosumab for the primary or secondary prevention of osteoporotic fragility fractures who do not meet the criteria specified in recommendations 1.1 or 1.2 should have the option to continue treatment until they and their clinician consider it appropriate to stop.
  T-score measures bone mineral density using central (hip and/or spine) dual-energy X-ray (DXA) scanning, and is expressed as the number of standard deviations (SD) below peak bone mineral density.# The technology 
Denosumab (Prolia, Amgen) is a monoclonal antibody that reduces osteoclast activity, and so reduces bone breakdown. Denosumab has a UK marketing authorisation for the treatment of osteoporosis in postmenopausal women at increased risk of fractures. The summary of product characteristics states in the indication that denosumab significantly reduces the risk of vertebral, non-vertebral and hip fractures.
The summary of product characteristics states that conditions associated with denosumab treatment include: urinary tract infection, upper respiratory tract infection, sciatica, cataracts, constipation, rash, pain in extremity and skin infections (predominantly cellulitis). However, there was no evidence of increased incidence of cataracts or diverticulitis in postmenopausal women with osteoporosis; these conditions occurred only in patients with prostate cancer. The summary of product characteristics states that osteonecrosis of the jaw has been reported in patients receiving denosumab or bisphosphonates, with most cases occurring in people with cancer, but some occurred in people with osteoporosis. For full details of side effects and contraindications, see the summary of product characteristics.
Denosumab is administered as a single subcutaneous injection into the thigh, abdomen or back of the arm. The recommended dosage is 60 mg once every 6 months.
The acquisition cost of denosumab is 183 for a 1 ml pre-filled syringe (60 mg per ml solution; excluding VAT, 'MIMS' September 2010 edition), which is equivalent to 366 for 1 year of treatment. Costs may vary in different settings because of negotiated procurement discounts.# The manufacturer's submission
The Appraisal Committee (appendix A) considered evidence submitted by the manufacturer of denosumab and a review of this submission by the Evidence Review Group (ERG; appendix B).
The manufacturer's submission presented data for clinical effectiveness from one main randomised trial, the FREEDOM (fracture reduction evaluation of denosumab in osteoporosis every 6 months) study. This multicentre, double-blind, placebo-controlled trial enrolled 7868 postmenopausal women aged 6090 years with T-scores of less than 2.5 SD and greater than 4.0 SD at lumbar spine, total hip, or both locations. The T-score measures bone mineral density using central (hip and/or spine) DXA scanning and is expressed as the number of standard deviations (SD) below the mean bone mineral densityof young, healthy adults of the same gender at their peak bone mass. Lower T-scores indicate lower bone mineral density. Women were randomly assigned to receive a subcutaneous injection of either 60 mg denosumab or placebo twice a year for 3 years. All participants also took daily calcium and vitamin D supplements.
The primary outcome was the incidence of new radiographically diagnosed vertebral fractures. Secondary outcomes were time to first non-vertebral fracture and time to first hip fracture. Health-related quality of life was assessed in terms of change from baseline in patient-reported outcomes using both the osteoporosis assessment questionnaire-short version (OPAQ-SV; physical function, emotional status and back pain score), and the EUROQOL-5D (EQ-5D) questionnaire.
The results of the FREEDOM study demonstrated that, based on the number of people who underwent spinal radiography at baseline and during at least one visit after baseline, the 36-month incidence of new radiographically diagnosed vertebral fractures was 2.3% (86 of 3702 women) in the denosumab group compared with 7.2% (264 of 3691 women) in the placebo group (relative risk  0.32, 95% confidence interval  0.26 to 0.41; p < 0.001). The reduction in risk was similar during each year of the trial. Similar reductions in incidence were seen for clinically diagnosed vertebral fractures (0.8% for denosumab versus 2.6% for placebo; hazard ratio  0.31, 95% CI 0.20 to 0.47; p < 0.001) and for multiple new radiographically diagnosed vertebral fractures (0.6% for denosumab versus 1.6% for placebo; RR 0.39, 95% CI 0.24 to 0.63; p < 0.001). Denosumab also reduced the risk of non-vertebral fracture (6.5% for denosumab versus 8.0% for placebo; HR 0.80, 95% CI 0.67 to 0.95; p = 0.01) and hip fracture (0.7% for denosumab versus 1.2% for placebo; HR 0.60, 95% CI 0.37 to 0.97; p = 0.04). The manufacturer stated that dropout rates were similar between groups and no imbalances were observed.
Health-related quality of life was assessed using the OPAQ-SV and EQ-5D questionnaire at baseline and every 6 months for 3 years. Among women who completed the study, completion rates for measures of health-related quality of life at year 3 were 83% for OPAQ-SV and 82% for EQ-5D. No significant differences were seen between treatment groups in measures of health-related quality of life at baseline compared with year 3, or between women without any fractures and those with incident clinical fractures. Changes from baseline to year 3 for each OPAQ-SV dimension and EQ-5D scores were positively correlated (all p < 0.0001).
A statistically significant difference was noted in skin infections, which occurred in 12 women receiving denosumab compared with one woman receiving placebo (p = 0.002). However, when all studies of denosumab were pooled in the manufacturer's meta-analysis, the overall incidences of adverse events, serious adverse events and adverse events leading to treatment withdrawal were generally similar between denosumab and placebo groups. Further safety data were available from 30 studies, giving a total of 14,000 patients, including 11,000 postmenopausal women with low bone density or osteoporosis, as well as people taking denosumab for preventing bone loss in prostate or breast cancer.
The manufacturer stated that it was mindful of the need for efficient use of NHS resources, and that, given the wide availability of generic oral bisphosphonates, denosumab was expected to be an option for women in whom oral bisphosphonates are unsuitable (reasons for unsuitability are that the woman is unable to comply with the special instructions for the administration of oral bisphosphonates, or has a contraindication to or is intolerant of oral bisphosphonates). Therefore denosumab was not expected to compete with oral bisphosphonates in clinical practice. In the absence of head-to-head clinical trials comparing denosumab with all relevant comparators (denosumab, strontium ranelate, raloxifene, teriparatide and zoledronate), the manufacturer carried out a random-effects meta-analysis of the relative risks (RRs) for all fracture endpoints directly comparing each treatment against placebo. The fracture incidence data (and RRs) for strontium ranelate for hip and wrist fracture were taken from the publication by Reginster et al. (2008) which reported 5-year data from the TROPOS study. As outlined in the table 1 below, the results of the manufacturer's meta-analysis showed that all treatments were associated with statistically significant decreases in the risk of morphometric vertebral fractures compared with placebo. Denosumab, strontium ranelate and zoledronate were associated with statistically significant decreases in the risk of clinical vertebral fractures, but raloxifene was not (no data were available for teriparatide). Similarly, denosumab, strontium ranelate, teriparatide and zoledronate were associated with statistically significant decreases in the risk of non-vertebral fractures, but raloxifene was not. Denosumab and zoledronate were associated with statistically significant decreases in the risk of hip fractures but strontium ranelate, and teriparatide were not (no data were available for raloxifene). None of the treatments were associated with a statistically significant decrease in the risk of wrist fracture.
Table 1 Manufacturer's direct comparison of each comparator with placebo from the random effects meta-analysis
Comparator
Clinically diagnosed vertebral fracture (relative risk )
Non-vertebral fractures (relative risk )
Hip fracture(relative risk )
Wrist fracture(relative risk )
Denosumab
(0.21 to 0.48)*
(0.69 to 0.96)*
(0.37 to 1.0)*
(0.64 to 1.1)
Zoledronate
(0.14 to 0.37)*
(0.65 to 0.87)*
(0.42 to 0.83)*
Raloxifene
(0.05 to 3.82)
(0.16 to 2.65)
Strontium ranelate
(0.50 to 0.84)*
(0.78 to 0.99)*
(0.67 to 1.2)
(0.73 to 1.31)
CI, confidence interval.
*Statistically significant (p  0.05).
The manufacturer's submission included a systematic review of the cost-effectiveness evidence for denosumab. The manufacturer carried out Markov cohort modelling to assess the cost effectiveness of denosumab against primary and secondary comparators. Primary comparators were strontium ranelate, raloxifene and no treatment (placebo). Secondary comparators were intravenous ibandronate, zoledronate and teriparatide. The manufacturer stated that denosumab is expected to be a treatment option for women with osteoporosis for whom oral bisphosphonates are unsuitable. Therefore, comparisons with oral bisphosphonates were not directly relevant to this appraisal and were included in appendices to the manufacturer's submission. The manufacturer stated that 71.6% of women receiving treatment for osteoporosis in England and Wales receive alendronate, 15.8% receive risedronate, 1.5% receive etidronate and 4.3% receive oral ibandronate, meaning that 93.2% of this population receive oral bisphosphonates (2009 figures). This means an estimated 6.8% of women receiving treatment for osteoporosis in England and Wales receive drugs other than oral bisphosphonates (2.8% strontium ranelate, 2.2% raloxifene, 0.6% intravenous ibandronate, 0.7% zoledronate, 0.2% calcitonin, 0.2% calcitriol and 0.1% teriparatide).
The manufacturer stated that persistence and compliance with oral bisphosphonates are poor because of the strict and complex dosing regimen and side effects of treatment. The manufacturer's submission stated that at least 42% of patients taking oral bisphosphonates stop within 1 year, and the median duration of treatment is estimated to be as low as 1.2 years. The manufacturer stated that few people (< 1%) permanently discontinued denosumab treatment because of treatment-related adverse events over 23 years in the FREEDOM study.
The model assessed the cost effectiveness of denosumab against the primary and secondary comparators for two separate cohorts. The first investigated the primary prevention of fragility fractures in women (70 years and over) with osteoporosis (T-score of 2.5 SD or below) for whom oral bisphosphonates are unsuitable. The second investigated the secondary prevention of subsequent fragility fractures in women (70 years and over) with osteoporosis (T-scores of 2.5 SD or below) and prior fragility fractures in whom oral bisphosphonates are unsuitable. The model had a cycle length of 6 months and a lifetime horizon (defined as until time of death or age of 100 years), including a half-cycle correction, with a treatment duration of 5 years.
The model included six discrete health states: well, hip fracture, clinically diagnosed vertebral fracture, wrist fracture, other types of fracture (pelvic, femur shaft, tibia, fibular, humerus, scapula, clavicle, rib or sternum), and death. It included two additional health states (post-hip fracture and post-vertebral fracture) to account for the long-term costs and effects associated with these fractures (no long-term costs or effects were assumed for women with wrist or other fractures). When a fracture occurred, women were modelled to remain in the respective fracture state for two cycles (1 year). After this period, women with a wrist fracture or other types of fracture were modelled to return to the well state. Women with a vertebral fracture or hip fracture were modelled to enter a post-fracture state. Women who had a vertebral fracture could no longer incur a wrist fracture or other type of osteoporotic fracture (other than a subsequent vertebral fracture or hip fracture). Women who had a hip fracture could only incur further hip fractures. The manufacturer's model was not a treatment-sequencing model because of the lack of clinical evidence for such use.
The manufacturer's base-case analysis assumed that women continued osteoporosis therapy for 5 years, and costs and quality-adjusted life years (QALYs) were tracked over the lifetime of the cohorts (consistent with economic modelling in 'Alendronate, etidronate, risedronate, raloxifene and strontium ranelate for the primary prevention of osteoporotic fragility fractures in postmenopausal women'  and 'Alendronate, etidronate, risedronate, raloxifene, strontium ranelate and teriparatide for the secondary prevention of osteoporotic fragility fractures in postmenopausal women' ). This assumption was examined in a sensitivity analysis.
Subgroup analysis was undertaken for women with and without prior fracture by age (5575, 5-year age bands) and T-score (between 2.5 to 4.0 SD). Sensitivity analysis assessed the effect on cost effectiveness of the presence or absence of additional independent clinical risk factors for fracture in women of 70 years of age, with and without prior fragility fractures. Sensitivity analysis also assessed the effects on cost effectiveness of differences in treatment persistence and compliance.
In the manufacturer's base-case analysis, fracture risks were estimated on the basis of epidemiological literature, and were based on three main elements: general population fracture risk, increased fracture risk associated with osteoporosis, and risk reduction attributed to treatment (if any). A systematic review of the literature was undertaken to identify appropriate UK studies or systematic reviews for all three model parameters. Age-specific fracture risks were estimated for women in the general population (using a study by Singer et al.  to estimate risk of wrist and hip fractures, and a study by Kanis et al.  to derive estimates for the incidence of clinically diagnosed vertebral and other fractures). Next, age-matched Z-scores (that is, the estimate of the number of SD below the mean bone mineral density of the general population for the patient's age and sex) were estimated for a cohort with osteoporosis using the National Health and Nutrition Examination Survey (NHANES) III database. Evidence from the systematic review was then used to attribute age-specific relative risks for the different types of fracture. Treatment was modelled to continue for 5 years by applying relative risks to the estimated baseline risks of fracture in the cohort with osteoporosis. An assumption was made that, on stopping treatment after 5 years, women would return in a linear fashion to baseline risk levels over 1 year (a return to baseline over 5 years was assumed in NICE technology appraisal guidance 160 and 161). The relative risks of fracture for each treatment for clinical vertebral, hip and wrist fractures were estimated from the manufacturer's direct comparison for each treatment against placebo if data were available. If evidence was not available for a comparator, the following explicit assumptions were made:
that for interventions without data for the relative risk of clinical vertebral fracture, this was equivalent to the relative risk of morphometric vertebral fracture
that the relative risk for interventions for which data for wrist and hip fractures were not available was 1.00.
that since no efficacy evidence was identified for intravenous ibandronate compared with placebo, efficacy was equivalent to that of oral ibandronate
that the relative risk for other fractures was 1.00 for all treatments, because 'other fracture' was not defined consistently across studies.
The model accounted for observed increases in the risk of mortality after fracture by applying relative risks for mortality obtained from a review of the literature. An increased risk was modelled for the first year and subsequent years after hip fracture or vertebral fracture. For other types of fracture, women were modelled to be at increased risk of mortality for 1 year only. The relative risks of mortality after all types of fracture were adjusted downwards to account for the observation that a proportion of mortality after fracture is explained by comorbidity. It was assumed that 30% of all mortality after all types of fracture is causally related, which is consistent with similar assumptions in NICE technology appraisal guidance 160 and 161.
The manufacturer's model also took into account persistence and compliance. Persistence is defined as the duration of time from start to end of therapy, and compliance is defined as conforming to the recommendations made by the provider with respect to timing, dosage and frequency of medication taking. Persistence and compliance were assumed to be 100% for the 5-year treatment period for all modelled treatments. Sensitivity analysis was carried out for oral therapies and denosumab.
Women completed the EQ-5D questionnaire in the FREEDOM study, but the number of fracture events with associated EQ-5D scores recorded was low and the trial design precluded assessment of health status immediately after fracture events. Therefore, evidence from the manufacturer's systematic review of the literature on health-related quality of life in osteoporosis was considered to be more meaningful and was applied in the economic analysis. The disutilities associated with fracture were obtained from a systematic review of the literature and applied to population norms in the form of utility multipliers. Utility loss associated with hip and vertebral fractures was modelled in a two-stage process, with a larger decrease in the first year after fracture and an ongoing but less severe utility loss in subsequent years. Utility multipliers for the first and subsequent years after hip fracture were obtained from a meta-analysis of studies using the EQ-5D responses. Utility loss associated with clinically diagnosed vertebral fracture was estimated separately for women managed in hospital and in primary care. The disutilities for women in hospital were derived from the EQ-5D scores of a cohort that were predominantly in hospital. The disutilities for women who were not in hospital were obtained from cohorts with prevalent morphometric fractures. Utility multipliers associated with wrist fracture were also obtained from the literature and applied in the model for 1 year after the event. Because of an absence of evidence, the same multiplier and the same approach were also used to model utility loss associated with other types of fractures. Finally, utility losses associated with selected adverse events were also included in the model.
Treatment costs and quality-of-life losses associated with wrist fracture or other types of fracture were modelled to last 1 year. Clinically diagnosed vertebral fractures and hip fractures were modelled to incur ongoing costs and loss of quality of life.
Costs of drug treatment were estimated using the 'British national formulary' (edition 58), with assumptions about the costs of administration and monitoring for the comparators. Fracture costs were estimated using hospital episode statistics for England and Wales in conjunction with the Department of Health's Healthcare Resource Group tariff; assumptions about the proportion of women treated in hospital, with and without surgery, for the different fracture types were informed by a combination of expert opinion, review of the literature and analysis of routine data. Costs associated with severe adverse events (such as gastrointestinal adverse events associated with oral therapies and cellulitis associated with denosumab) were included. Other types of adverse events associated with denosumab and its comparators were not included.
The results of the manufacturer's base-case analysis (pairwise comparisons) for the primary comparators showed that, for primary prevention, the incremental cost-effectiveness ratios (ICERs) for denosumab were 29,223 per QALY gained compared with no treatment and 9289 per QALY gained compared with raloxifene, and denosumab dominated strontium ranelate (that is, denosumab was less costly and more effective). For secondary prevention, the ICERs for denosumab were 12,381 per QALY gained compared with no treatment, 2046 per QALY gained compared with raloxifene, and denosumab dominated strontium ranelate. ICERs compared with no treatment for primary prevention were 74,239 per QALY gained for raloxifene and 102,592 per QALY gained for strontium ranelate. ICERs compared with no treatment for secondary prevention were 24,524 per QALY gained for raloxifene and 37,123 per QALY for strontium ranelate.
The results of the manufacturer's base-case analysis (pairwise comparisons) for the secondary comparators showed that denosumab was the lowest-cost treatment. For primary prevention, the ICERs for the other treatments compared with denosumab were 70,900 per QALY gained for zoledronate, 772,424 per QALY gained for teriparatide, and denosumab dominated ibandronate. For secondary prevention, the ICERs for the other treatments compared with denosumab were 29,029 per QALY gained for zoledronate, 451,269 per QALY gained for teriparatide, and denosumab dominated ibandronate.
The manufacturer presented a subgroup analysis to demonstrate how the cost effectiveness of denosumab varied when using different treatment cut-offs (that is, all women with a T-score at or below 2.5, 3, 3.5 SD and so on). The manufacturer provided further subgroup analyses for women with and without prior fracture by age and T-score. The results of the manufacturer's subgroup analyses showed that the cost effectiveness of denosumab improved as age increases and as T-score decreases, and with the presence of a prior fragility fracture. For primary prevention, in circumstances in which none of the treatments appraised by NICE are recommended, and oral bisphosphonates are unsuitable, the ICER for denosumab compared with no treatment varied between 19,313 and 71,319 per QALY gained. In circumstances in which strontium ranelate is recommended for primary prevention, denosumab dominated strontium ranelate (that is, denosumab was more effective and less costly). For secondary prevention, in circumstances in which no treatment is currently recommended in the NHS, the ICER for denosumab compared with no treatment varied between 12,289 and 22,957 per QALY gained. In circumstances in which strontium ranelate is recommended for secondary prevention, denosumab dominated strontium ranelate, and in circumstances in which raloxifene is recommended for secondary prevention, denosumab dominated raloxifene or had an ICER of 2046 per QALY gained.
The manufacturer also provided a subgroup analysis using the FRAX algorithm (an internet-based tool developed by the World Health Organization to calculate a 10-year absolute risk of fracture). This showed how cost effectiveness varied depending on T-score and the presence or absence of independent clinical risk factors for fracture. The results demonstrated that the presence of independent clinical risk factors for fracture, particularly rheumatoid arthritis, also improved the cost effectiveness of denosumab compared with the primary comparators (strontium ranelate, raloxifene and no treatment).
The manufacturer conducted a range of deterministic and probabilistic sensitivity analyses. The results of the deterministic sensitivity analyses showed that alterations to most key parameters had limited impact on comparisons of denosumab with raloxifene, strontium ranelate and no treatment. The impact on comparisons with intravenous ibandronate, zoledronate and teriparatide were most sensitive to changes in assumptions about the cost of denosumab administration. The manufacturer carried out sensitivity analyses that assumed one administration of denosumab in secondary care per year. Under this scenario, the ICER for denosumab compared with no treatment rose to 36,185 per QALY gained in women with no prior fragility fracture, and to 15,720 per QALY gained in women with a prior fragility fracture. This change led to zoledronate dominating denosumab in women with and without a prior fragility fracture. Following a request from the ERG, the manufacturer also carried out a sensitivity analysis in which denosumab treatment was assumed to be started in secondary care and thereafter delivered in general practice. This analysis showed that the additional cost associated with initiating treatment with denosumab in a secondary care setting had a marginal impact on the cost-effectiveness of denosumab compared with both primary and secondary comparators. The ERG also requested that the manufacturer provided further analysis assuming equal efficacy of denosumab and zoledronate for the prevention of wrist fractures. This analysis showed that the ICER for denosumab was moderately sensitive to assumptions about the relative efficacy of the two drugs for the prevention of wrist fractures.
After consultation, the manufacturer carried out additional sensitivity analyses on the long-term effects of fractures on mortality and nursing home care using the conservative assumption that nursing home admission was zero. These analyses showed no substantial impact on the cost-effectiveness results with the ICER for denosumab compared with strontium ranelate going from denosumab being dominant (that is, denosumab was less costly and more effective than strontium ranelate) to 2040 per QALY gained for primary prevention, and denosumab remaining dominant for secondary prevention. The ICER for denosumab compared with raloxifene went from 9289 (11,135 costs, 8.0 QALYs) to 12,438 per QALY gained for primary prevention, and from 2,046 (13,543 costs, 7.9 QALYs) to 5,120 for secondary prevention.
The results of the manufacturer's probabilistic sensitivity analysis showed that denosumab had approximately 50% probability of being considered cost effective at a threshold of 30,000 per QALY gained compared with the primary comparators (strontium ranelate, raloxifene and no treatment) in the base-case population of women aged 70 years with a T-score at or below 2.5 SD for primary prevention. The probability for secondary prevention was 90%. Against the secondary comparators (ibandronate, zoledronate and teriparatide), denosumab had a 60% probability of being considered cost effective at a threshold of 30,000 per QALY gained in the base-case population of women aged 70 years without prior fracture. The probability in women with prior fracture was 70%.
The ERG considered that the evidence of clinical effectiveness presented in the manufacturer's submission was derived from a large high-quality trial of adequate duration. The ERG stated that it did not consider the evidence presented in the manufacturer's submission on the effects of drugs on bone mineral density to be relevant because fracture data were available for all drugs. The ERG also noted that the data for morphometric vertebral fractures were not relevant, and so were not used in the modelling.
The ERG noted that the results for the direct comparison of strontium ranelate with placebo (RR for hip fracture of 0.89 and RR for non-vertebral fracture of 0.88) were similar to the meta-analysis provided in NICE technology appraisal 160 (RR for hip fracture of 0.85 and RR for all non-vertebral fractures of 0.84), which provided some confidence in the results. The ERG expressed concern about the relevant comparator for denosumab (see 3.29) and the methodology of the meta-regression to determine whether mean age and bone mineral density were associated with different effects of treatments.
The ERG noted that the manufacturer provided multiple comparisons of cost effectiveness using a high-quality validated model that took into account a wide range of costs, such as short-term drug costs and long-term nursing home costs, and that the analysis met the NICE reference case. The ERG considered that the appendices to the manufacturer's submission also provided very detailed accounts of underlying model assumptions and sensitivity analyses.
The ERG identified several issues with the manufacturer's economic model, specifically:
the choice of comparator
cost assumptions for denosumab
the validity of assumptions used for modelling utilities, costs, persistence and compliance
variations in cost effectiveness in subgroups of the cohort modelled
-mission of underlying fracture risk estimates from the probabilistic sensitivity analysis
treatment setting and administration of denosumab.
First, the ERG believed that zoledronate should be a key comparator. The manufacturer's submission did not consider zoledronate or intravenous ibandronate to be primary comparators for denosumab because they are used by only 0.7% and 0.6% of currently treated women respectively (according to Intercontinental Marketing Services data), and neither comparator had been appraised by NICE. However, the ERG stated that intravenous ibandronate and zoledronate are licensed and used routinely in UK secondary care for treating osteoporosis in postmenopausal women. The ERG noted that intravenous ibandronate and zoledronate were similar in effectiveness but that intravenous ibandronate was given more frequently than zoledronate, and would be associated with greater administration costs. Therefore, given both its effectiveness and the same method of intravenous administration, zoledronate was a key comparator in the ERG's view.
The second issue raised by the ERG was that the relative cost effectiveness of denosumab compared with zoledronate depended on assumptions made about administration costs. The manufacturer assumed that denosumab would be given twice a year in general practice at the average cost of two standard visits to a GP, whereas zoledronate was assumed to be given once a year in hospital clinics (with some monitoring incorporated into the visit). The ERG believed that this approach made denosumab much less costly than zoledronate. Therefore, the ERG believed that, given the similar effectiveness of denosumab and zoledronate, the cost-effectiveness comparison depended largely on the relative costs used in the model. The ERG carried out additional exploratory analyses assuming that denosumab was given entirely in secondary care, which demonstrated that for primary prevention the ICER for denosumab compared with no treatment was 40,627 per QALY gained. For primary prevention, the ICER for denosumab compared with raloxifene was 25,743 per QALY gained and for denosumab compared with strontium ranelate was 15,866 per QALY gained. For secondary prevention, the ICER for denosumab compared with no treatment was 17,851 per QALY gained, the ICER for denosumab compared with raloxifene was 12,171 per QALY gained, and the ICER for denosumab compared with strontium ranelate was 6606 per QALY gained.
After comments on the appraisal consultation document,  which reported a change in cost of zoledronate and that an alternative relative risk could be used for the effect of zoledronate on wrist fracture, the ERG were requested to carry out additional sensitivity analyses. These demonstrated that the change in the cost of zoledronate (which reduced from 283.74 to 266.72 in January 2010) resulted in ICERs for zoledronate compared with denosumab decreasing from 70,900 to 55,885 per QALY gained for primary prevention, and from 29,029 to 22,966 per QALY gained for secondary prevention. Additional sensitivity analyses were also carried out using an alternate relative risk value of 0.81 for risk of wrist fracture for zoledronate (instead of 1.0 in the manufacturer's base case, and 0.84 in the ERGs original sensitivity analyses). The results showed that the ICER for zoledronate compared with denosumab decreased from 70,900 to 58,764 per QALY gained for primary prevention, and from 29,029 to 24,454 for secondary prevention. The ERG carried out analyses using the manufacturer's assumptions in their economic model by combining both the above zoledronate changes simultaneously. These resulted in an ICER for zoledronate compared with denosumab of 44,804 per QALY gained for primary prevention, and 18,606 per QALY gained for secondary prevention.
The ERG identified that a simplifying assumption was used for transitions in the model. Women experiencing a vertebral fracture could no longer experience a wrist fracture or other type of fracture (apart from a clinical vertebral fracture or hip fracture). After a hip fracture, women could no longer experience any type of fracture other than a hip fracture. The ERG believed that this assumption was unrealistic because experience of a hip fracture or clinical vertebral fracture would put women at higher risk of further fracture. However, the extent of the effect of these assumptions on the cost-effectiveness estimates was unclear.
The ERG noted that in the manufacturer's base-case analysis, the assumption that fracture risk would return linearly to baseline levels over the course of 1 year after stopping treatment was conservative and would favour oral therapies. Persistence and compliance were assumed to be 100% for all treatments in the base-case analysis, which was also a conservative assumption. The ERG noted that after initial administration of denosumab, both compliance and persistence would be 100% for 6 months. However, in the long term, persistence with denosumab therapy may be less than 100%. The manufacturer carried out sensitivity analyses that examined variations in persistence for oral therapies and denosumab.
The ERG noted that the manufacturer's quality-of-life review methodology and the primary studies included in the review suggested that suitable utility multipliers were applied in the model. However, many of the multipliers were derived from observational time-series studies without independent control groups and therefore did not control for all potential confounding factors. The ERG noted that costs and utility losses associated with wrist fractures and other types of fracture were assumed to last for 1 year, whereas hip fractures and clinical vertebral fractures were modelled to incur ongoing costs and utility losses. The ERG also noted that utility loss relative to population norms remained constant in the second and subsequent years after hip fracture or vertebral fracture. This assumption may have slightly overestimated utility loss associated with hip and vertebral fracture if the observed trend towards improved quality of life in the second year after fracture continued in subsequent years.
The ERG noted that the manufacturer's ICERs varied substantially within subgroups of the cohorts, and that the appropriate comparator also varied by subgroup according to existing NICE guidance. Furthermore, neither raloxifene or strontium ranelate compared favourably with no treatment (ICERs of 74,239 and 102,592 per QALY gained respectively for 70-year-old women with a T-score of 2.5 SD and no prior fragility fracture, and 24,524 and 37,123 per QALY gained respectively for those with a prior fragility fracture), which is consistent with the modelling in NICE technology appraisal guidance 160 and 161. The ERG expressed the view that demonstrating cost effectiveness against these comparators did not allow the conclusion that denosumab is cost effective. The ERG also believed that, for the comparison between denosumab and zoledronate, there was uncertainty about the costs of administering these two drugs and their relative efficacy for the prevention of wrist fracture.
The manufacturer conducted a range of deterministic and probabilistic sensitivity analyses. The ERG noted that an important omission from the probabilistic sensitivity analysis was the underlying estimates of fracture risk. The manufacturer stated that data limitation meant that distributions could not be estimated for these parameters. The ERG believed that this would have the effect of overestimating the probability of denosumab being considered cost effective at different payment thresholds. It also noted that deterministic sensitivity analysis showed that the cost-effectiveness estimates were sensitive to underlying fracture risk. Following consultation the manufacturer carried out additional sensitivity analyses in which beta distributions were assigned to baseline fracture incidence based on an assumed sample size of 10,000. Probabilistic sensitivity analysis showed that for denosumab compared with no treatment the ICER was 30,422 per QALY gained which was similar to the deterministic ICER of 29,233 per QALY gained.
The ERG had concerns about the treatment setting and administration of denosumab in the model. The subcutaneous injection of denosumab is simple and could be carried out by a general practitioner, a practice nurse or the woman herself. However, the ERG believed that denosumab treatment would probably not be started in general practice because it is a new biological agent that has effects on other body systems (including the immune system), and that long-term adverse events could not be ruled out. The ERG stated that it would expect at least one outpatient visit to be needed and, in many cases, continued hospital follow-up would be necessary. Additionally, if follow-up was partly or mainly in general practice, the ERG believed that administration of denosumab would probably not be provided in primary care as part of general medical services, but would be regarded as an enhanced service for which an additional payment would be negotiated (the size of which is currently unknown, but may be greater than the manufacturer's assumption of the average cost of two visits to a GP per patient per year). Therefore, the marginal costs per patient of administering denosumab in primary care may be greater than the average cost of two visits to a GP per patient per year as presented in the manufacturer's model.
The ERG noted that although denosumab could be self-administered by the woman, the average age of women taking medication for the prevention of fracture in the General Practice Research Database dataset was 71.4 years, and many would be older. Such women might not be able to give themselves a subcutaneous injection because of poor eyesight, poor manual dexterity or cognitive impairment. The oldest age groups also have the highest proportion of women treated with oral bisphosphonates. Furthermore, training women to self-administer denosumab might not be regarded as worthwhile because they would have to visit a general practice to obtain the pre-filled pen injection device, and after 6 months some may have forgotten how to administer it (which is unlikely to occur with drugs given daily, such as teriparatide). The ERG also expressed the view that an equality issue exists for women who have had a stroke in the past and who are at increased risk of falls and fracture, together with bone loss because of reduced mobility. Such women might have difficulty swallowing or standing to take oral bisphosphonates, and therefore, intravenous or subcutaneous drugs may be more suitable.
Full details of all the evidence are in the manufacturer's submission and the ERG report.# Consideration of the evidence
The Appraisal Committee reviewed the data available for the clinical and cost effectiveness of denosumab, having considered evidence on the nature of osteoporotic fractures and the value placed on the benefits of denosumab by postmenopausal women with the condition, those who represent them, and clinical specialists. It also took into account the effective use of NHS resources.
The Committee discussed the clinical need of postmenopausal women for the prevention of osteoporotic fragility fractures. It heard from the clinical specialists that the main aim of primary prevention is the opportunistic identification of postmenopausal women who are at risk of osteoporotic fragility fractures, and the aim of secondary prevention is to provide the most effective treatment for women who have already had an osteoporotic fragility fracture and are at risk of further fractures.
The Committee heard from the clinical specialists that current UK clinical practice is to start treatment with oral bisphosphonates (first with alendronate and then either risedronate or etidronate if alendronate is unsuitable). These treatments are not suitable for all women because some women are unable to comply with the special instructions for the administration of oral bisphosphonates, or have a contraindication to or are intolerant of oral bisphosphonates. The Committee also heard from the clinical specialists that women for whom oral bisphosphonates are unsuitable receive either no treatment or strontium ranelate for primary prevention (as set out in NICE technology appraisal guidance 160), or no treatment, strontium ranelate or raloxifene for secondary prevention (as set out in NICE technology appraisal guidance 161). The clinical specialists stated that the management of osteoporosis usually takes place in primary care (both strontium ranelate and raloxifene are given in primary care). Women who have severe osteoporosis may receive more potent agents such as zoledronate or intravenous ibandronate but there is limited capacity for treatment in secondary care because of the need for day-case facilities for these intravenous treatments. The patient experts stated that some women who cannot take or cannot tolerate oral bisphosphonates have a preference for strontium ranelate or raloxifene as they do not like the intravenous infusion used for zoledronate treatment, whereas others prefer the convenience of a 12-monthly intravenous infusion (zoledronate) over taking oral treatments daily (strontium ranelate and raloxifene). The Committee accepted that the great majority of treatment for the primary and secondary prevention of osteoporotic fragility fractures is provided in primary care. It also accepted that women for whom oral therapies are unsuitable or who have severe osteoporosis may receive more potent agents such as zoledronate or intravenous ibandronate in secondary care and that teriparatide is also used for secondary prevention when women are unable to take other therapies. The Committee concluded that the relevant comparators for primary prevention are no treatment and strontium ranelate, and for secondary prevention are no treatment, strontium ranelate and raloxifene, because both the administration and supervision of strontium ranelate and raloxifene are organised in primary care. The Committee also concluded that potential comparators for denosumab are zoledronate (for severe osteoporosis) and teriparatide (for women who have sustained a clinically apparent osteoporotic fracture and who are defined by age, T score and number of osteoporotic fractures and who are unable to take all oral bisphosphonates, strontium and raloxifene, as defined in NICE Technology Appraisal 161).
The Committee was aware that the licensed indication for denosumab is for the treatment of osteoporosis in postmenopausal women at increased risk of fractures. The Committee noted that the manufacturer's decision problem focused on postmenopausal women diagnosed with osteoporosis for whom oral bisphosphonates are unsuitable. The manufacturer stated that denosumab was not expected to compete with oral bisphosphonates in clinical practice, given the wide availability of generic oral bisphosphonates in the UK and the need for efficient use of NHS resources. The Committee also noted that the manufacturer did provide an analysis of denosumab compared with oral bisphosphonates for completeness. It accepted that it was reasonable to base its considerations on women for whom oral bisphosphonates are unsuitable and the subsequent discussion focused on this population only.
The Committee heard from the clinical specialists that denosumab is a monoclonal antibody that reduces osteoclast activity and hence reduces bone breakdown, that it is the first drug of its class, and that its biological mechanism of action results in targeted therapy with fewer adverse events than other treatments. The Committee considered that a treatment administered by subcutaneous injection once every 6 months, such as denosumab, offers an advantage because it may improve adherence with therapy, particularly for women who have problems swallowing or standing to take oral bisphosphonates. The Committee also accepted evidence from the patient experts that many women stop taking oral bisphosphonates because of adverse events, and often do not go back to their GP. Therefore a 6-monthly subcutaneous injection of denosumab could provide women with a pre-arranged opportunity to discuss their treatment and any adverse events with a healthcare professional, and this support may improve compliance and persistence with treatment.

# Clinical effectiveness
The Committee accepted that the clinical-effectiveness evidence presented in the manufacturer's submission was derived from a large, high-quality trial of adequate duration (FREEDOM) that studied treatment with denosumab compared with placebo. The Committee noted that, because the FREEDOM study did not provide a head-to-head comparison of denosumab against all relevant comparators, the manufacturer carried out a random-effects meta-analysis to obtain direct estimates for each treatment compared with placebo (denosumab, strontium ranelate, raloxifene, teriparatide and zoledronate).
The Committee noted that the FREEDOM trial showed a statistically significant 68% reduction in the relative risk p < 0.001) of the 36-month incidence of new radiographically diagnosed vertebral fractures. The Committee also noted that denosumab significantly reduced the risk of non-vertebral fracture (HR 0.80; 95% CI 0.67 to 0.95; relative reduction of 20%) and of hip fracture (HR 0.60, 95% CI 0.37 to 0.97; relative reduction of 40%). The Committee concluded that the evidence from the FREEDOM trial demonstrated that denosumab was effective in reducing the risk of fracture in postmenopausal women compared with placebo.
The Committee discussed the meta-analysis that was undertaken by the manufacturer to obtain direct estimates for each treatment compared with placebo. However, the Committee noted the lack of a direct comparison of denosumab with active comparators. It was, therefore, unable to make a conclusion about the relative clinical effectiveness of denosumab, but was satisfied with the evidence on the direct estimates for each treatment compared with placebo and concluded that the methods used in the meta-analysis were sufficiently robust for use in the economic analysis.
The Committee discussed the adverse events experienced by postmenopausal women receiving denosumab for the prevention of osteoporotic fragility fractures. The Committee noted that in the FREEDOM trial treatment with denosumab was associated with fewer adverse events than placebo. The patient experts stated that the main concern women have about treatment for osteoporosis is the duration for which therapies are taken and whether they will experience adverse events over a long period of time. The Committee heard that there may be a risk of infection associated with denosumab treatment; however, it accepted evidence from the clinical specialists that this risk was low and that if substantial evidence became available, it may be necessary to assess women with severe infections before considering the use of denosumab. The Committee noted that studies of denosumab for other indications have shown that treatment may be associated with osteonecrosis of the jaw, but it was satisfied with the clinical specialists' statement that there was no evidence of this from the clinical studies of denosumab in women with osteoporosis. The clinical specialists confirmed that 14,000 women have received denosumab and that it was well tolerated. The Committee concluded that the available clinical evidence on the adverse effects associated with denosumab indicated that it was a well tolerated treatment for the prevention of osteoporotic fragility fractures in postmenopausal women.

# Cost effectiveness
The Committee considered the manufacturer's economic model, and the critique and exploratory sensitivity analyses performed by the ERG. It noted that the manufacturer used a Markov economic model to evaluate the cost effectiveness of denosumab compared with a range of comparators, and that the clinical data were derived from the manufacturer's direct comparison of each comparator with placebo in their random-effects meta-analysis. The Committee considered that the methods used in the analysis were robust.
The Committee considered the ERG's concerns about a number of aspects of the economic model, such as the long-term effects of fractures on mortality, the setting where denosumab is likely to be given, and the associated administration and monitoring costs modelled. The Committee also discussed issues raised from consultation on the appraisal consultation document: the use of quality of life data from the FREEDOM study in the economic model, the use of the FRAX tool to estimate fracture risk, the cost and setting of care for women receiving denosumab, the cost of zoledronate, the relative risk for wrist fracture for zoledronate, and the relative risk for hip fracture with strontium ranelate.
The Committee considered the long-term effects of fractures on mortality and the need for nursing home care. It noted that the manufacturer assumed that 30% of observed mortality for all fracture types is causally related to osteoporotic fracture, and this estimate was varied to 100% in sensitivity analyses. The Committee was aware that the manufacturer's model also assumed that women were at increased risk of entry into nursing home care following hip fracture and that the lowest cost of private residential care was applied. The Committee noted that some women may already be in a nursing home, and some may also be self-funding their nursing home care. After consultation, the manufacturer carried out additional sensitivity analyses, using the assumption that nursing home admission was zero. The Committee concluded that the long-term effects of fractures on mortality and nursing home care had only a minor impact on the cost-effectiveness estimates for denosumab.
The Committee noted that the manufacturer's model assumed treatment duration of 5 years, and that the FREEDOM study was of 3 years' duration. The Committee accepted that the 5-year treatment duration assumption was appropriate and reflected clinical practice. The Committee noted that the manufacturer's model assumed that women would return in a linear fashion to baseline fracture risk levels over 1 year after treatment stops. It considered whether the return to baseline fracture risk should be different according to treatment type, and heard from the ERG that the duration of benefit in terms of fracture risk (as opposed to bone mineral density) is unknown after cessation of osteoporosis treatments. The Committee concluded that there was little evidence on the duration of effect on fracture risk for osteoporosis treatments and that this was an area of uncertainty.
The Committee discussed health-related quality-of-life benefits and utility values in the economic model. The Committee noted that no significant differences were seen in health-related quality of life between the denosumab and placebo arms of the FREEDOM trial. The manufacturer justified the omission from the economic model of EQ-5D data from FREEDOM on the grounds that the number of fracture events with associated EQ-5D scores was low and there was relative infrequency of health-related quality of life measurement. The ERG accepted that the lack of statistical difference in EQ-5D scores between the denosumab and placebo arms of the trial was explained by the above factors (low number of fracture events with associated EQ-5D scores and relative infrequency of health-related quality of life measurement), rather than an adverse effect of denosumab masking the health benefit of fracture prevention. The ERG stated that omitting the FREEDOM trial EQ-5D data from the economic analysis was justified given the quality and depth of the manufacturer's systematic review of health-related quality-of-life data. The Committee concluded that the manufacturer's approach to modelling health-related quality of life was acceptable.
The Committee considered that the manufacturer's model allowed sensitivity analyses to be carried out using the FRAX tool. The Committee accepted that FRAX is a potentially useful tool in clinical practice, but it was mindful that the tool is presently unvalidated. The Committee was not persuaded that recommendations about treatment should be based on absolute risk as calculated using FRAX and that the stepped approach of assessing fracture risk is needed to ensure the effective allocation of NHS resources. This is because absolute fracture risk is the total risk for all fracture sites, but different fracture sites have different impacts on quality of life, costs and mortality. Therefore, cost effectiveness is dependent on the contribution from each fracture site to the total fracture risk. The Committee concluded that using a combination of T-score, age and a number of independent clinical risk factors for fracture remained more appropriate for defining treatment recommendations in this appraisal.
The Committee considered the key drivers of cost effectiveness and noted that alterations to most key parameters in the manufacturer's sensitivity analyses had limited impact on comparisons of denosumab with raloxifene, strontium ranelate and no treatment. Comparisons with ibandronate, zoledronate and teriparatide were most sensitive to changes in the assumptions about the cost of administering denosumab. When the manufacturer increased the cost of administering denosumab (by assuming that one administration per year would be delivered in secondary care), this increased the ICER for denosumab compared with no treatment from 29,200 to 36,200 per QALY gained for primary prevention, and from 12,400 to 15,700 per QALY gained for secondary prevention. The Committee noted that given the similar cost and efficacy of denosumab and zoledronate, changes to this assumption also resulted in zoledronate dominating denosumab (that is, zoledronate was less costly and more effective than denosumab) for both primary and secondary prevention. The Committee concluded that with the exception of administration costs for denosumab, alterations to most key parameters had limited impact on comparisons between denosumab and the primary and secondary comparators.
The Committee noted the ERG's view that administration of denosumab may not be provided in primary care. However, the clinical specialists stated that there is no reason why denosumab should only be used in secondary care. The clinical specialists highlighted that because denosumab is a new biological agent they expected that, initially, treatment would be started in secondary care, but with follow-up almost exclusively in primary care (except for women with severe osteoporosis, who may be followed up in secondary care in line with current UK clinical practice). The Committee discussed whether administering denosumab would be part of general medical services or whether it would be regarded as an enhanced service for which an additional payment would be negotiated, and it noted the comments received during consultation on the ACD. The clinical specialists stated that women would not need to go through a screening process before starting treatment with denosumab, and that women receiving denosumab are not likely to be at high risk of side effects and so follow-up in secondary care would not be necessary. The clinical specialists also stated that although denosumab is a biological agent and also has effects on the immune system, it is specifically targeted for regulating bone cells. The clinical specialists, therefore, thought that the potential safety concerns associated with other biological agents (such as those targeting tumour necrosis factor) may not be applicable to denosumab. The clinical specialists stated that because treatment with denosumab would not involve substantial additional activities to standard practice in managing osteoporosis, it would probably be provided as part of general medical services. The Committee accepted the views of the clinical specialists that there were no specific safety concerns around the use of denosumab and that follow-up in secondary care would not be necessary. Therefore, it was not persuaded to alter its opinion that denosumab is likely to be provided as part of general medical services in primary care. The Committee concluded that while treatment with denosumab may be started in secondary care, it would be subsequently delivered almost exclusively in primary care. The relatively small proportion of women with severe osteoporosis would continue to be followed-up in secondary care, in line with current UK clinical practice.
After consultation on the appraisal consultation document, the Committee discussed comments that outlined a change in the cost of zoledronate and that an alternative relative risk could be used for the effect of zoledronate on wrist fracture. The ERG was requested to carry out exploratory analyses that showed that denosumab was less effective and less costly than zoledronate. The Committee had already concluded that although treatment with denosumab may be started in secondary care, it will be subsequently delivered almost exclusively in primary care, unlike the administration of zoledronate, use of which will remain in secondary care. As the Committee regarded the main comparators for denosumab to be those treatments delivered in primary care when oral bisphosphonates were unsuitable (no treatment, strontium ranelate, raloxifene), it did not regard these issues to be central to the decision problem.
The Committee, therefore, accepted that the most likely cost-effectiveness estimates would lie between the manufacturer's base case (using primary care assumptions) and the ERG's additional analyses (using secondary care assumptions), and that the most plausible ICERs were likely to be closer to the manufacturer's base case estimates given that care would mostly be in the primary setting.
The Committee considered the relative risks for hip fracture that were used in the manufacturer's meta-analysis. The Committee was aware that NICE recommendations on strontium ranelate are due to be reconsidered following a court of appeal judgement related to assumptions about relative risk of hip fracture. It noted that the figure that the manufacturer used as the 5-year relative risk of hip fracture for strontium ranelate was 0.89 compared with placebo and this was derived from the published TROPOS study, but that alternative relative risk figures of 0.64 (obtained over 3 years) or 0.57 (obtained over 5 years) for the effect of strontium ranelate on hip fracture were suggested during consultation. These alternative, and lower relative risk values, were based on a post-hoc subgroup analysis in the TROPOS study of women over the age of 74 with a T score of 2.4. The Committee also noted a suggestion by the manufacturer of denosumab that if a subgroup relative risk figure for hip fracture was to be used for one treatment in comparative analysis, then similar subgroup figures should also be used for denosumab for this and other outcomes.
The Committee noted that when using a relative risk of hip fracture of 0.89 for strontium ranelate in the manufacturer's base case, denosumab was dominant compared with strontium ranelate for both primary and secondary prevention. The Committee heard from the ERG that exploratory analyses applying the relative risk estimate of 0.64 over the modelled 5-year treatment period in the manufacturer's model resulted in a base-case ICER of 10,200 per QALY gained for denosumab compared with strontium ranelate for primary prevention and an ICER of 5100 per QALY gained for secondary prevention. When the relative risk estimate of 0.57 for strontium ranelate was applied over the modelled 5-year treatment period in the manufacturer's model, this resulted in an ICER of 16,300 per QALY gained for denosumab compared with strontium ranelate for primary prevention and an ICER of 8600 per QALY gained for denosumab compared with strontium ranelate for secondary prevention. The Committee considered these exploratory analyses and concluded that it did not need to make a decision on which relative risk for strontium ranelate was the most appropriate one to apply, because for any of the suggested relative risk values for hip fracture for strontium ranelate, the ICERs for denosumab compared with strontium ranelate fell within a range that was still considered to be a cost effective use of NHS resources.
The Committee noted that for the primary prevention of osteoporotic fragility fractures, denosumab dominated strontium ranelate in the manufacturer's base case, and the ICER was 15,900 per QALY gained for denosumab compared with strontium ranelate in the ERG's additional analyses (assuming denosumab is always given in secondary care). The Committee noted that in NICE technology appraisal guidance 160, strontium ranelate is recommended for postmenopausal women:
who are unable to comply with the special instructions for the administration of alendronate and either risedronate or etidronate, or have a contraindication to or are intolerant of alendronate and either risedronate or etidronate and
who also have a combination of T-score, age and number of independent clinical risk factors for fracture as indicated in the following table.
T-scores (SD) at (or below) which strontium ranelate is recommended when alendronate and either risedronate or etidronate cannot be taken
Age (years)
Number of independent clinical risk factors for fracture
-r older
  Treatment with strontium ranelate is not recommended.
For the purposes of technology appraisal guidance 160, independent clinical risk factors for fracture are parental history of hip fracture, alcohol intake of 4 or more units per day, and rheumatoid arthritis. The Committee accepted that the above were appropriate when making recommendations on treatment with denosumab for the prevention of primary and secondary osteoporotic fractures. The Committee was mindful that the clinical effectiveness evidence was based on a meta-analysis of each treatment compared with placebo rather than a direct comparison of denosumab with active comparators, and of the areas of uncertainty it had discussed in the economic modelling. However it was satisfied with the robustness of the clinical evidence and, when taken together with the low ICERs presented, the Committee concluded that, as an option, denosumab was a cost-effective use of NHS resources for the primary prevention of osteoporotic fragility fractures only for postmenopausal women at increased risk of fractures for whom oral bisphosphonates are unsuitable, and who have the same level of fracture risk as described in the recommendations for strontium ranelate in NICE technology appraisal guidance 160.
The Committee was aware that for women for whom oral bisphosphonates are unsuitable and who do not currently fulfil the criteria for treatment with strontium ranelate, there is no treatment currently recommended by NICE for the primary prevention of osteoporotic fragility fractures. The Committee noted that the ICER for denosumab compared with no treatment was 29,200 per QALY gained in the manufacturer's base-case analysis, and this increased to 40,600 per QALY gained in the ERG's additional analyses. It concluded that the ICER for the base-case population (women 70 years and over with a T-score of 2.5 SD or below) for denosumab compared with no treatment was likely to lie within this range. The Committee explored the results for subgroups of women at higher risk by age and T-score, for whom oral bisphosphonates are unsuitable and in circumstances in which none of the treatments appraised by NICE are recommended. The ICERs for denosumab compared with no treatment from the manufacturers model varied between 19,300 and 71,300 per QALY gained. The Committee agreed that the most plausible ICERs were likely to be higher, based on the ERG's amended assumptions. Taking into account the uncertainties around the long-term effects of fractures on mortality and nursing home care, and the setting where denosumab is likely to be given, the Committee concluded that denosumab was not a cost-effective use of NHS resources for women for whom oral bisphosphonates are unsuitable and the treatments appraised by NICE are not recommended.
For the secondary prevention of osteoporotic fragility fractures, the Committee noted that the ICER for denosumab compared with no treatment in women for whom oral bisphosphonates are unsuitable was 12,400 per QALY gained in the manufacturer's base-case analysis, which increased to 17,900 per QALY gained in the ERG's additional analyses. Denosumab dominated raloxifene or had an ICER of 2000 per QALY gained in the manufacturer's base-case analysis, which increased to 12,200 per QALY gained in the ERG's additional analyses. The cost-effectiveness results for denosumab compared with strontium ranelate ranged from strontium ranelate being dominated by denosumab in the manufacturer's base-case analysis to an ICER of 6600 per QALY gained in the ERG's exploratory analyses. The Committee also noted the results of the subgroup analysis by age and T-score for women for whom oral bisphosphonates are unsuitable and in circumstances in which the treatments appraised by NICE are not recommended, in which the ICER for denosumab compared with no treatment varied between 12,289 and 22,957 per QALY gained. The Committee noted that teriparatide is recommended in NICE technology appraisal guidance 161 as a treatment option for the secondary prevention of osteoporotic fragility fractures in women with severe osteoporosis for whom oral bisphosphonates and strontium ranelate are unsuitable. The Committee noted that in the manufacturer's base-case analysis, denosumab was slightly less effective and much less costly than teriparatide.
The Committee concluded that treatment with denosumab was a cost-effective use of NHS resources and may be an option for the secondary prevention of osteoporotic fragility fractures in women for whom oral bisphosphonates are unsuitable (that is, in women who are unable to comply with the special instructions for administering alendronate and either risedronate or etidronate, or have an intolerance of, or a contraindication to, those treatments). The Committee also concluded that denosumab may provide an alternative treatment option that would be a cost-effective use of NHS resources for women who are eligible for treatment with teriparatide as defined in NICE technology appraisal guidance 161.
The Committee considered whether its recommendations were associated with any potential issues related to equality. Following consultation on the appraisal consultation document, the Committee was aware that one area of potential discrimination was the primary prevention of osteoporotic fractures in postmenopausal women with swallowing problems as a result of disabling stroke disease, who would otherwise be eligible for treatment with oral bisphosphonates, but do not fulfil the criteria for denosumab or other treatments. The Committee concluded that its considerations were based on the population of postmenopausal women for whom oral bisphosphonates are unsuitable which included women with and without a disability.

# Summary of Appraisal Committee's key conclusions
TA204 STA
Appraisal Title: Denosumab for the prevention of osteoporotic fractures in postmenopausal women
FAD section
Key conclusion
Denosumab is recommended as a treatment option for the primary prevention of osteoporotic fragility fractures only in postmenopausal women at increased risk of fractures:
who are unable to comply with the special instructions for administering alendronate and either risedronate or etidronate, or have an intolerance of or a contraindication to those treatments and
who have a combination of T-score, age and number of independent clinical risk factors for fracture (see section 1.3)  as indicated in the following table.
T-scores (SD) at (or below) which denosumab is recommended when alendronate and either risedronate or etidronate are unsuitable
Age (years)
Number of independent clinical risk factors for fracture
-r older
  Treatment with denosumab is not recommended.
Denosumab is recommended as a treatment option for the secondary prevention of osteoporotic fragility fractures only in postmenopausal women at increased risk of fractures who are unable to comply with the special instructions for administering alendronate and either risedronate or etidronate, or have an intolerance or contraindication to those treatments.
For the purposes of this guidance, independent clinical risk factors for fracture are parental history of hip fracture, alcohol intake of 4 or more units per day, and rheumatoid arthritis.
The Committee accepted that there was good quality evidence to support the clinical effectiveness of denosumab compared with placebo and that denosumab would be a clinically effective alternative to existing treatment options for the prevention of osteoporotic fragility fractures in postmenopausal women.
The Committee concluded that treatment with denosumab was a cost-effective use of NHS resources for the primary prevention of osteoporotic fragility fractures only in postmenopausal women at increased risk of fractures for whom oral bisphosphonates are unsuitable, and who have the same level of fracture risk as described in the recommendations of NICE technology appraisal 160 for strontium ranelate.
The Committee concluded that treatment with denosumab was a cost-effective use of NHS resources for the secondary prevention of osteoporotic fragility fractures in postmenopausal women at increased risk of fractures for whom oral bisphosphonates are unsuitable.
Current practice
Clinical need of patients including the availability of alternative treatments
The main aim of primary prevention is the opportunistic identification of postmenopausal women who are at risk of osteoporotic fragility fractures, and the aim of secondary prevention is to provide the most effective treatment for women who have already had an osteoporotic fragility fracture and are at risk of further fractures.
The Committee accepted that current UK clinical practice is to start treatment with oral bisphosphonates (first with alendronate and then either risedronate or etidronate if alendronate is unsuitable), but that these are not suitable for all women. Reasons for unsuitability are that the woman is unable to comply with the special instructions for the administration of oral bisphosphonates, or has a contraindication to or is intolerant of oral bisphosphonates. Women for whom oral bisphosphonates are unsuitable receive either no treatment or strontium ranelate for primary prevention (as set out in NICE technology appraisal guidance 160), or strontium ranelate or raloxifene for secondary prevention (as set out in NICE technology appraisal guidance 161). Treatment usually takes place in primary care (both strontium ranelate and raloxifene are given in primary care). Women who have severe osteoporosis may receive more potent agents such as zoledronate or intravenous ibandronate.
The Committee also noted that zoledronate is sometimes used in secondary care for the prevention of fractures, and that teriparatide is also used for secondary prevention when women are unable to take other therapies.
The technology
Proposed benefits of the technology
How innovative is the technology in its potential to make a significant and substantial impact on health-related benefits?
The Committee considered that a treatment that is administered by subcutaneous injection once every 6 months, such as denosumab, offers an advantage because it may improve adherence with therapy, particularly for women who have problems swallowing or standing to take oral bisphosphonates.
The Committee noted that denosumab is a monoclonal antibody that reduces osteoclast activity and hence reduces bone breakdown. It also heard from the clinical specialists that it is the first drug in its class, and that its targeted biological mechanism of action results in fewer adverse events than other treatments.
What is the position of the treatment in the pathway of care for the condition?
The Committee noted that the manufacturer's decision problem focused on postmenopausal women diagnosed with osteoporosis for whom oral bisphosphonates are unsuitable, and that the manufacturer stated that denosumab was not expected to compete with oral bisphosphonates in clinical practice, given the wide availability of generic oral bisphosphonates in the UK. The Committee also noted that the manufacturer did provide an analysis of denosumab compared with oral bisphosphonates for completeness. It accepted that it was reasonable to base its considerations on women for whom oral bisphosphonates are unsuitable and the subsequent discussion focused on this population only.
Adverse effects
The Committee heard that there may be a risk of infection associated with denosumab treatment, however it accepted evidence from the clinical specialists that this risk was low and that if substantial evidence became available, it may be necessary to assess women with severe infections before considering the use of denosumab.
The Committee concluded that the available clinical evidence on the adverse effects associated with denosumab indicated that it was a well tolerated treatment for the prevention of osteoporotic fragility fractures in postmenopausal women.
Evidence for clinical effectiveness
Availability, nature and quality of evidence
The Committee considered that the clinical effectiveness evidence presented in the manufacturer's submission was derived from a large, high-quality trial of adequate duration (FREEDOM). Because the FREEDOM study did not provide a head-to-head comparison of denosumab against all relevant comparators, the manufacturer carried out a random-effects meta-analysis to obtain direct estimates for each treatment compared with placebo.
Relevance to general  clinical practice in the NHS
The Committee accepted that denosumab is not expected to compete with oral bisphosphonates in clinical practice, given the wide availability of generic oral bisphosphonates in the UK as stated by the manufacturer.
Uncertainties generated by the evidence
There was no head-to-head clinical trial evidence comparing denosumab with active relevant comparators (the comparator in the FREEDOM trial was placebo). The manufacturer carried out a random-effects meta-analysis to obtain direct estimates for each treatment compared with placebo. However, the Committee noted the lack of a direct comparison of denosumab with active comparators. It was, therefore, unable to make a conclusion about the relative clinical effectiveness of denosumab, but was satisfied with the evidence on the direct estimates for each treatment compared with placebo and concluded that the methods used in the meta-analysis were sufficiently robust for use in the economic analysis.
The Committee heard that there may be a risk of infection associated with denosumab treatment; however, it accepted evidence from the clinical specialists that this risk was low and that if substantial evidence became available, it may be necessary to assess women with severe infections before considering the use of denosumab.
Are there any clinically relevant subgroups for which there is evidence of differential effectiveness
Subgroups by T-score, age and clinical risk factors were explored in the economic model and considered by the Appraisal Committee (see below).
Estimate of the size of the clinical effectiveness including strength of supporting evidence
The Committee concluded that the evidence from the FREEDOM trial demonstrated that denosumab was effective in reducing the risk of fracture in postmenopausal women compared with placebo.
The Committee discussed the meta-analysis that was undertaken by the manufacturer to obtain direct estimates for each treatment compared with placebo. However, the Committee noted the lack of a direct comparison of denosumab with active comparators. It was therefore unable to make a conclusion about the relative clinical effectiveness of denosumab, but was satisfied with the evidence on the direct estimates for each treatment compared with placebo and concluded that the methods used in the meta-analysis were sufficiently robust for use in the economic analysis.
The Committee accepted that there was good evidence to support the clinical effectiveness of denosumab compared with placebo and that denosumab it would be a clinically effective alternative to existing treatment options for the prevention of osteoporotic fragility fractures in postmenopausal women.
For details of treatment effect size see relative risks reported in sections 3.3, 3.6 and 4.7.
Evidence for cost effectiveness
Availability and nature of evidence
The Committee accepted that the manufacturer used a Markov economic model to evaluate the cost effectiveness of denosumab compared with a range of comparators (split into primary and secondary comparators). The clinical data were derived from the manufacturer's direct comparison of each comparator with placebo from their random-effects meta-analysis.
Uncertainties around and plausibility of assumptions and inputs in the economic model
The Committee was mindful of the ERG's concerns around a number of aspects of the economic model, such as the long term effects of fractures on mortality, the setting where denosumab is likely to be given, and the associated administration and monitoring costs modelled. The Committee also discussed issues raised from consultation on the appraisal consultation document: the use of the FRAX tool to estimate fracture risk, the cost and setting of care for women receiving denosumab, the cost of zoledronate, the relative risk for hip fracture with strontium ranelate.
The Committee concluded that the long term effects of fractures on mortality and nursing home care did not have a substantial impact on the cost effectiveness estimates for denosumab.
The Committee noted that the manufacturer's model assumed treatment duration of 5 years, and that the FREEDOM study was of 3 years' duration. The Committee accepted that the 5-year treatment duration assumption was appropriate and reflected clinical practice. The Committee heard from the ERG that the duration of benefit in terms of fracture risk (as opposed to bone mineral density) is unknown after cessation of osteoporosis treatments. The Committee concluded that there was little evidence on the duration of effect on fracture risk for osteoporosis treatments and that this was an area of uncertainty.
The Committee considered that the manufacturer's model allowed sensitivity analyses to be carried out using the FRAX tool and accepted that it is potentially useful in clinical practice. However, it was not persuaded that recommendations about treatment should be based on absolute risk as calculated using FRAX and that the stepped approach of assessing fracture risk is needed to ensure the effective allocation of NHS resources. Therefore, the Committee concluded that using a combination of T-score, age and a number of independent clinical risk factors for fracture remained more appropriate for defining treatment recommendations in this appraisal.
The Committee concluded that with the exception of administration costs for denosumab, alterations to most key parameters had limited impact on comparisons between denosumab and the primary and secondary comparators.
The Committee considered the change in cost of zoledronate and that an alternative relative risk could be used for the effect of zoledronate on wrist fracture. The Committee concluded that although treatment with denosumab may be started in secondary care, it will be subsequently delivered almost exclusively in primary care, unlike the administration of zoledronate, use of which will remain in a secondary care setting. As the Committee regarded the main comparators for denosumab to be those treatments delivered in primary care when oral bisphosphonates were unsuitable, it did not regard these issues to be central to the decision problem.
The Committee considered the relative risks for hip fracture that were used in the manufacturer's meta-analysis. The Committee was aware that NICE recommendations on strontium ranelate are due to be reconsidered following a court of appeal judgement related to assumptions about relative risk of hip fracture. It noted that the figure that the manufacturer used as the 5-year relative risk of hip fracture for strontium ranelate was 0.89 compared with placebo and this was derived from the published TROPOS study, but that alternative relative risk figures of 0.64 (obtained over 3 years) or 0.57 (obtained over 5 years) for the effect of strontium ranelate on hip fracture were suggested during consultation.
The Committee noted that when using a relative risk of hip fracture of 0.89 for strontium ranelate in the manufacturer's base case, denosumab was dominant compared with strontium ranelate for both primary and secondary prevention. The Committee heard from the ERG that exploratory analyses applying the relative risk estimate of 0.64 over the modelled 5-year treatment period in the manufacturer's model resulted in a base-case ICER of 10,200 per QALY gained for denosumab compared with strontium ranelate for primary prevention and an ICER of 5100 per QALY gained for secondary prevention. When the relative risk estimate of 0.57 for strontium ranelate was applied over the modelled 5-year treatment period in the manufacturer's model, this resulted in an ICER of 16,300 per QALY gained for denosumab compared with strontium ranelate for primary prevention and an ICER of 8600 per QALY gained for denosumab compared with strontium ranelate for secondary prevention. The Committee considered these exploratory analyses and concluded that it did not need to make a decision on which relative risk for strontium ranelate was the most appropriate one to apply, because for any of the suggested relative risk values for hip fracture for strontium ranelate, the ICERs for denosumab compared with strontium ranelate fell within a range that was still considered to be a cost effective use of NHS resources.
Incorporation of health-related quality of life benefits and utility values
Have any potential significant and substantial health-related benefits been identified that were not included in the economic model, and how have they been considered?
The Committee noted that no significant differences were seen in health-related quality of life between the denosumab and placebo arms of FREEDOM.
The ERG stated that the omission of the FREEDOM trial EQ-5D data from the economic analysis was justified given the quality and depth of the manufacturer's systematic review of health-related quality of life data. The Committee concluded that the manufacturer's approach to modelling health-related quality of life was acceptable.
Are there specific groups of people for whom the technology is particularly cost-effective?
The Committee noted that for the primary prevention of osteoporotic fragility fractures, denosumab dominated strontium ranelate in the manufacturer's base case, and the ICER was 15,900 per QALY gained for denosumab compared with strontium ranelate in the ERG's additional analyses (assuming denosumab is always given in secondary care). The Committee noted that in NICE technology appraisal guidance 160, strontium ranelate is recommended for postmenopausal women:
- who are unable to comply with the special instructions for the administration of alendronate and either risedronate or etidronate, or have a contraindication to or are intolerant of alendronate and either risedronate or etidronate and
- who also have a combination of T-score, age and number of independent clinical risk factors for fracture is as indicated in the following table.
T-scores (SD) at (or below) which strontium ranelate is recommended when alendronate and either risedronate or etidronate cannot be taken
Age (years)
Number of independent clinical risk factors for fracture
-r older
  Treatment with strontium ranelate is not recommended.
- For the purposes of technology appraisal guidance 160, independent clinical risk factors for fracture are parental history of hip fracture, alcohol intake of 4 or more units per day, and rheumatoid arthritis.
The Committee accepted that the above were appropriate when making recommendations on treatment with denosumab for the prevention of primary and secondary osteoporotic fractures. The Committee was mindful that the clinical effectiveness evidence was based on a meta-analysis of each treatment compared with placebo rather than with active comparators. However, it was satisfied that the robustness of the clinical evidence and, when taken together with the low ICERs presented, the Committee concluded that for the primary prevention of osteoporotic fragility fractures, denosumab was a cost-effective use of NHS resources as a treatment option only for postmenopausal women at increased risk of fractures for whom oral bisphosphonates are unsuitable, and who have the same level of fracture risk as described in the recommendations of NICE technology appraisal 160 for strontium ranelate.
For the secondary prevention of osteoporotic fragility fractures, the Committee also noted the results of the subgroup analysis by age and T-score, for women for whom oral bisphosphonates are unsuitable and in circumstances in which the treatments appraised by NICE are not recommended, in which the ICER for denosumab compared with no treatment varied between 12,289 and 22,957 per QALY gained. The Committee concluded that treatment with denosumab was considered to be a cost-effective use of NHS resources for the secondary prevention of osteoporotic fragility fractures in women for whom oral bisphosphonates are unsuitable.
What are the key drivers of cost effectiveness?
The Committee concluded that with the exception of administration costs for denosumab, alterations to most key parameters had limited impact on comparisons between denosumab and the primary and secondary comparators.
Most likely cost-effectiveness estimate (given as an ICER)
The Committee accepted that the most likely cost-effectiveness estimates would lie between the manufacturer's base case (using primary care assumptions) and the ERG's additional analyses (using secondary care assumptions), and that the most plausible ICERs were likely to be closer to the manufacturer's base case estimates, given that care would mostly be in the primary setting.
The Committee noted that for the primary prevention of osteoporotic fragility fractures, denosumab dominated strontium ranelate in the manufacturer's base case, and the ICER was 15,900 per QALY gained for denosumab compared with strontium ranelate in the ERG's additional analyses (assuming denosumab is always given in secondary care). The Committee concluded that for the primary prevention of osteoporotic fragility fractures, denosumab was a cost-effective use of NHS resources as a treatment option only for postmenopausal women at increased risk of fractures for whom oral bisphosphonates are unsuitable, and who have the same level of fracture risk as described in the recommendations of NICE technology appraisal 160 for strontium ranelate.
The Committee was aware that for women for whom oral bisphosphonates are unsuitable and who do not currently fulfil the criteria for treatment with strontium ranelate, no treatment is currently recommended by NICE for the primary prevention of osteoporotic fragility fractures. The Committee noted that the ICER for denosumab compared with no treatment was 29,200 per QALY gained in the manufacturer's base-case analysis, and this increased to 40,600 per QALY gained in the ERG's additional analyses. It concluded that the ICER for the base-case population (women 70 years and over with T-score of 2.5 SD or below) for denosumab compared with no treatment was likely to lie within this range.
For the secondary prevention of osteoporotic fragility fractures, the Committee noted that the ICER for denosumab compared with no treatment in women for whom oral bisphosphonates are unsuitable was 12,400 per QALY gained in the manufacturer's base-case analysis, which increased to 17,900 per QALY gained in the ERG's additional analyses. Denosumab dominated raloxifene or had an ICER of 2000 per QALY gained (age 70, T-score 2.5) in the manufacturer's base-case analysis, which increased to 12,200 per QALY gained in the ERG's additional analyses. The ICER for denosumab compared with strontium ranelate ranged from strontium ranelate being dominated by denosumab in the manufacturer's base-case analysis to 6600 per QALY gained in the ERG's additional analyses.
The Committee concluded that treatment with denosumab was considered to be a cost-effective use of NHS resources for the secondary prevention of osteoporotic fragility fractures in women for whom oral bisphosphonates are unsuitable. The Committee noted that in the manufacturer's base-case analysis, denosumab was slightly less effective and much less costly than teriparatide. Therefore the Committee concluded that denosumab may provide an alternative treatment option that would be a cost-effective use of NHS resources for women who are eligible for treatment with teriparatide as defined in NICE technology appraisal guidance 161.
Additional factors taken into account
Patient access schemes (PPRS)
Not applicable. No patient access scheme was submitted.
End of life considerations
Not applicable to this disease area.
Equalities considerations
Following consultation on the appraisal consultation document, the Committee was aware that one area of potential discrimination was the primary prevention of osteoporotic fractures in postmenopausal women with swallowing problems as a result of disabling stroke disease, who would otherwise be eligible for treatment with oral bisphosphonates, but do not fulfil the criteria for denosumab or other treatments. The Committee concluded that its considerations were based on the population of postmenopausal women for whom oral bisphosphonates are unsuitable which included women with and without a disability.
  T-score relates to the measurement of bone mineral density using central (hip and/or spine) DXA scanning, and is expressed as the number of standard deviations (SD) below peak bone mineral density.
  Treatment with denosumab is not recommended.
  Treatment with strontium ranelate is not recommended.# Related NICE guidance
Alendronate, etidronate, risedronate, raloxifene and strontium ranelate for the primary prevention of osteoporotic fragility fractures in postmenopausal women (amended).NICE technology appraisal guidance 160 (2008; amended 2010).
Alendronate, etidronate, risedronate, raloxifene, strontium ranelate and teriparatide for the secondary prevention of osteoporotic fragility fractures in postmenopausal women (amended).NICE technology appraisal guidance 161 (2008; amended 2010).# Review of guidance
The guidance on this technology will be considered for review at the same time that NICE technology appraisal guidance 160 and 161 (2008; amended 2010) are considered for review. The Guidance Executive will decide whether the technology should be reviewed based on information gathered by NICE, and in consultation with consultees and commentators.
Andrew DillonChief ExecutiveOctober 2010# Changes after publication
February 2014: implementation section updated to clarify that denosumab is recommended as an option for preventing osteoporotic fractures in postmenopausal women. Additional minor maintenance update also carried out.
March 2012: minor maintenance# About this guidance
NICE technology appraisal guidance is about the use of new and existing medicines and treatments in the NHS in England and Wales.
This guidance was developed using the NICE single technology appraisal process.
We have produced a summary of this guidance for patients and carers. Tools to help you put the guidance into practice and information about the evidence it is based on are also available.
Your responsibility
This guidance represents the views of NICE and was arrived at after careful consideration of the evidence available. Healthcare professionals are expected to take it fully into account when exercising their clinical judgement. However, the guidance does not override the individual responsibility of healthcare professionals to make decisions appropriate to the circumstances of the individual patient, in consultation with the patient and/or guardian or carer.
Implementation of this guidance is the responsibility of local commissioners and/or providers. Commissioners and providers are reminded that it is their responsibility to implement the guidance, in their local context, in light of their duties to avoid unlawful discrimination and to have regard to promoting equality of opportunity. Nothing in this guidance should be interpreted in a way which would be inconsistent with compliance with those duties.
Copyright
 National Institute for Health and Clinical Excellence 2010. All rights reserved. NICE copyright material can be downloaded for private research and study, and may be reproduced for educational and not-for-profit purposes. No reproduction by or for commercial organisations, or for commercial purposes, is allowed without the written permission of NICE.