The Role and Limitations of 18-Fluoro-2-deoxy-d-glucose Positron Emission Tomography -LRB- FDG-PET -RRB- Scan and Computerized Tomography -LRB- CT -RRB- in Restaging Patients with Hepatic Colorectal Metastases Following Neoadjuvant Chemotherapy : Comparison with Operative and Pathological Findings Background Recent data confirmed the importance of 18-fluoro-2-deoxy-d-glucose positron emission tomography -LRB- FDG-PET -RRB- in the selection of patients with colorectal hepatic metastases for surgery . Neoadjuvant chemotherapy before hepatic resection in selected cases may improve outcome . The influence of chemotherapy on the sensitivity of FDG-PET and CT in detecting liver metastases is not known . Introduction The liver is the most common , and often the only , site of distant metastases from colorectal cancer -LRB- CRC -RRB- .1 Hepatic resection is the only effective therapy for a subset of patients with CRC metastatic to the liver , and is associated with 5-year survival rates ranging from 25 to 40 % .2 -- 5 From 60 to 65 % of patients will , however , develop recurrent tumors after hepatic resection , indicating that they had harbored unrecognized intra - or extrahepatic tumor foci at the time of liver resection .6 Moreover , several studies report unresectable disease in 40 -- 70 % of patients that undergo laparotomy for liver resection .2,7,8 These data indicate that better patient selection is needed to avoid unnecessary operations . There are several potential ways of improving patient selection , one of which is the administration of neoadjuvant therapy followed by reevaluation and better preoperative staging . Positron emission tomography with the glucose analog 18-fluoro-2-deoxy-d-glucose -LRB- FDG-PET -RRB- is a sensitive diagnostic tool for the detection of colorectal metastases . Approximately 25 % of patients are discovered to have new intra - or extrahepatic tumors on FDG-PET performed after standard imaging .9 -- 13 Screening with FDG-PET before hepatic resection for CRC significantly improves the survival rates of resected patients , probably by improving patient selection .14 The role of neoadjuvant chemotherapy to down-stage nonresectable liver metastases and to improve outcome following hepatic resection of resectable liver metastases is an evolving concept , but one that is not yet established . With the recent application of new chemotherapeutic agents , such as irinotecan , oxaliplatin , and bevacizumab , improved response rates can be achieved and the use of these agents in the neoadjuvant setting would appear to be especially relevant for patients with nonresectable disease or patients with high risk of recurrence .15 -- 18 The aim of our study was to examine the effect of neoadjuvant chemotherapy for hepatic colorectal metastases on CT and FDG-PET/CT findings and to define the role of these imaging techniques in this setting . To do so , we compared CT and FDG-PET/CT findings with histopathological reports . Patients and Methods Patients Patients with colorectal liver metastases were assigned to receive either an immediate liver resection -LRB- group 1 -RRB- or neoadjuvant chemotherapy -LRB- group 2 -RRB- . The criteria for neoadjuvant treatment were : Nonresectable tumors due to size , location , and number and assessment of the surgical team that complete -LRB- R0 -RRB- resection was not technically possible.High risk of recurrence according to the Memorial Sloan -- Kettering Cancer Center -LRB- MSKCC -RRB- clinical risk score to assess risk of recurrence .19 Specifically , patients with two or more risk factors -LSB- number of metastases > 1 , disease-free survival < 12 months , carcinoembryonic antigen -LRB- CEA -RRB- levels > 200 ng/ml , metastases from the colonic tumor to regional lymph nodes , size of the largest metastases > 5 cm -RSB- were assigned to neoadjuvant treatment.Presence of extrahepatic disease.Oncologist 's preference -- this applied to patients with MSKCC > 2 that were referred from other hospitals for immediate surgery . The decision not to administer neoadjuvant therapy was not necessarily in agreement with our policy.Patient 's preference -- patients who refused neoadjuvant therapy were assigned to immediate surgery when feasible . Neoadjuvant Chemotherapy Treatment consisted of a neoadjuvant chemotherapeutic combination of 5-fluorouracil , leucovorin , and either oxaliplatin -LRB- FOLFOX -RRB- or irinotecan -LRB- FOLFIRI -RRB- . Seventeen patients -LRB- 35 % -RRB- were also given bevacizumab . Most of the group 2 patients were given neoadjuvant irinotecan unless they were enrolled on a multicenter study whose protocol consisted of the administration of neoadjuvant oxaliplatin . Staging Before undergoing neoadjuvant chemotherapy , all group 2 patients underwent a triphasic contrast-enhanced CT scan , and a FDG-PET/CT was performed in 30 -LRB- 62.5 % -RRB- of them . All 75 patients in group 1 and group 2 underwent FDG-PET/CT and abdominal CT before liver surgery . The time interval between the last course of chemotherapy and the FDG-PET/CT scan was at least 2 weeks , and surgical exploration took place within 1 month following the FDG-PET/CT scan in most of the cases . Because we used an integrated PET/CT technique , precise anatomical localization could be achieved and confirmed with the standard triphasic abdominal CT findings . PET/CT The patients were asked to fast for at least 4 h before undergoing PET/CT . Earlier lab tests had shown that they all had glucose levels < 150 mg % . The patients received an intravenous injection of 370 -- 666 MBq -LRB- 10 -- 18 mCi -RRB- of 18F-FDG . Data acquisitions by an integrated PET/CT system -LRB- Discovery LS ; GE Medical Systems , Milwaukee , WI , USA -RRB- were performed within 60 -- 120 min after injection . Iodinated oral contrast material was given to opacify loops of the bowel on the CT image . Data acquisition was as follows : CT scanning was performed first , from the head to the pelvic floor , with 140 kV , 80 mA , a tube rotation time of 0.5 s , a pitch of 6 , and a 5-mm section thickness , which was matched to the PET section thickness . Immediately after CT scanning , a PET emission scan that covered the identical transverse field of view was obtained . Acquisition time was 5 min per table position . PET image data sets were reconstructed iteratively by applying the CT data for attenuation correction , and coregistered images were displayed on a workstation -LRB- Xeleris , Elgems , Haifa , Israel -RRB- . Studies of all patients were retrieved and read in consensus by two experts -LRB- U.M. and E.E.-S . -RRB- . All suspected sites of metastatic disease showing an increased FDG uptake were recorded . The location of hepatic lesions was recorded according to the Couinaud segmental classification . Hepatectomy All patients without extensive extrahepatic disease underwent surgical exploration and intraoperative ultrasound -LRB- IOUS -RRB- . Resections of all metastatic sites were performed by either anatomic or R0 nonanatomic resection , with a tendency toward maximal parenchymal preservation with nonanatomic resections . Complete radiological response to neoadjuvant chemotherapy was defined as the complete resolution of all metastatic sites according to the CT and PET-CT . In these cases , careful palpation and IOUS were performed in search of remaining tumor or scarring . When there was no evidence of either , the tumor sites were resected according to the findings on the original imaging -LRB- i.e. , before any response to neoadjuvant treatment -RRB- . Detection of Hepatic Metastases To define the sensitivity of CT and FDG-PET/CT for liver metastases , imaging results were compared with the presence and size of liver lesions as demonstrated and measured by histopathological reports . Results Patients Between June 2002 and September 2005 , 75 patients with 155 suspected metastatic lesions from a primary CRC underwent hepatic resection in our department . Group 1 included 27 patients with 33 lesions who underwent immediate liver resection and group 2 included 48 patients with 122 lesions who first received neoadjuvant chemotherapy before subsequently undergoing liver resection . The patient 's profiles are outlined in Table 1 . Table 2 lists the operative procedures that were performed in the two groups . Table 1Study Patients ' Profiles Group 1 -LRB- n = 27 -RRB- Group 2 -LRB- n = 48 -RRB- P valueSex ratio -LRB- F/M -RRB- 0.500.920.22 Mean age , years -LRB- std deviation -RRB- 66 -LRB- 9.8 -RRB- 61.25 -LRB- 10.9 -RRB- 0.06 Site Colon9 -LRB- 71 % -RRB- 32 -LRB- 66 % -RRB- 0.74 Rectum8 -LRB- 29 % -RRB- 16 -LRB- 33 % -RRB- LN metastases -LRB- Duke 's > B in colonic specimen -RRB- 81.5 % 82 % 0.73 No . of liver tumors -LRB- mean -RRB- -LRB- std deviation -RRB- 1.19 -LRB- 0.4 -RRB- 2.52 -LRB- 1.9 -RRB- 0.0001 Max tumor diameter -LRB- largest -RRB- -LRB- std deviation -RRB- 3.53 cm -LRB- 2.84 -RRB- 3.9 cm -LRB- 1.84 -RRB- 0.49 Extrahepatic disease -LRB- no . of patients -RRB- 790.56 Prior liver resection461Mean MSKCC risk score -LRB- range -RRB- 1.82 -LRB- 0 -- 4 -RRB- 2.48 -LRB- 2 -- 5 -RRB- 0.003 Group 1 , immediate hepatic resection ; Group 2 , hepatic resection following neoadjuvant chemotherapyLN = lymph nodeTable 2Operative ProceduresOperative procedure -LRB- no . of patients ; lesions -RRB- Group 1 -LRB- n = 27 -RRB- Group 2 -LRB- n = 48 -RRB- Right hepatic lobectomy58Left hepatic lobectomy44Central hepatectomy03Right trisegmentectomy10Nonanatomic resections1529Left lat segmentectomy22Explorative laparotomy -LRB- no resection -RRB- 02Group 1 , immediate hepatic resection ; Group 2 , hepatic resection following neoadjuvant chemotherapy Detection of Hepatic Metastases The overall findings , the sensitivity , specificity , and accuracy of triphasic contrast-enhanced CT and FDG-PET/CT in the detection of viable liver metastases compared to the pathological results are presented in Table 3 . FDG-PET and CT had a statistically significant higher sensitivity in detecting liver metastases in patients who did not receive chemotherapy compared to patients who received chemotherapy -LRB- Table 3 -RRB- . Statistical analysis also revealed that triphasic contrast-enhanced CT had a higher sensitivity than PET/CT in detecting colorectal metastasis following neoadjuvant treatment -LRB- 65.3 vs 49 % , respectively , P < 0.0001 -RRB- , but not in patients who did not receive neoadjuvant therapy -LRB- 87.5 vs 93.3 % , P = 0.625 -RRB- . Table 3FDG-PET and CT -- Comparison With Pathological Results Group 1 n = 33Group 2 n = 122P valuePET TP2948 True negative -LRB- complete response -RRB- -- 20 FP24 FN250 Sensitivity93 .3 % 49 % < 0.0001 Specificity -- 83.3 % CT TP2864 True negative -LRB- complete response -RRB- -- 18 FP16 FN434 Sensitivity87 .5 % 65.3 % 0.038 Specificity -- 75 % Group 1 , immediate hepatic resection ; group 2 , hepatic resection following neoadjuvant chemotherapy Four of the six false-positive -LRB- FP -RRB- results on FDG-PET involved patients who had previously undergone hepatic resection . These lesions were discovered on follow-up FDG-PET/CT . Uptake was observed along the resection site , and these patients underwent nonanatomic liver resections for suspected locally recurrent lesions . Pathologic evaluation failed to reveal any tumor cells . The positive predictive value of FDG-PET/CT for metastasis recurrence in the resection site was only 33 % , and specificity was 60 % . Sensitivity of FDG-PET in the detection of colorectal metastasis correlated with the size of the metastasis -LRB- Table 4 -RRB- . Average size of the metastases in the two groups was 33.9 mm -LRB- standard deviation 19 -RRB- in group 1 and 18.9 mm -LRB- standard deviation 19 -RRB- in group 2 , P < 0.0001 . Table 4Sensitivity of FDG-PET : Correlation With Tumor SizeTumor size < 1 cm1 -- 3 cm > 3 cmGroup 1 sensitivity -LRB- total no . of lesions -RRB- 33 % -LRB- n = 3 -RRB- 100 % -LRB- n = 15 -RRB- 92 % -LRB- n = 13 -RRB- Group 2 sensitivity -LRB- total no . of lesions -RRB- 17 % -LRB- n = 35 -RRB- 78 % -LRB- n = 41 -RRB- 100 % -LRB- n = 22 -RRB- Group 1 , immediate hepatic resection ; group 2 , hepatic resection following neoadjuvant chemotherapy We also compared the sensitivity of CT and FDG-PET for patients who received FOLFIRI or FOLFOX -LRB- n = 31 -RRB- with patients who received the same regimen plus bevacizumab -LRB- n = 17 -RRB- . The results are outlined in Table 5 . We found that the sensitivity of FDG-PET , but not of CT , was lower in patients who received bevacizumab , although the difference did not reach statistical significance . Table 5FDG-PET and CT in Patients who Received Chemotherapy With or Without Bevacizumab : Comparison With Pathological Results Bevacizumab − Bevacizumab + P valuePET TP2919 True negative -LRB- complete response -RRB- 173 FP22 FN2030 Sensitivity59 % 39 % 0.068 CT TP3331 True negative -LRB- complete response -RRB- 135 FP60 FN1618 Sensitivity67 % 63 % 0.9 Detection of Extrahepatic Metastases In group 1 , there were one FP result for extrahepatic disease -LRB- suspected recurrence in colonic anastomosis , abdominal wall -RRB- , one true-positive -LRB- TP -RRB- result -LRB- recurrence in mesocolic lymph nodes -RRB- , and one false negative -LRB- FN -RRB- result -LRB- in a patient with peritoneal metastases -RRB- . In group 2 , there was one FP result -LRB- for suspected peritoneal metastasis -RRB- , three TP results -LRB- recurrence in paraaortic lymph nodes and solitary lung metastasis -RRB- , and two FN results -LRB- for peritoneal metastases -RRB- . Discussion The role of neoadjuvant chemotherapy followed by hepatectomy for colorectal liver metastases has not yet been clearly established . New chemotherapeutic agents , including irinotecan , oxaliplatin , and the biologic agent bevacizumab , have yielded improved response rates in the treatment of advanced CRC . These agents may have a potential role in the neoadjuvant setting for down-staging both nonresectable disease to resectability15 ,16 and resectable disease , probably mostly for patients with high risk of recurrence .17 Our policy is to administer neoadjuvant treatment to patients with nonresectable disease , those with extrahepatic disease , and those with resectable disease who have two or more risk factors according to the MSKCC clinical risk score .19 One of the theoretical benefits of neoadjuvant treatment is that patients who develop additional extrahepatic or intrahepatic metastases during this time period are spared a futile major operative procedure . Accurate staging before the beginning of neoadjuvant treatment and restaging following the treatment are crucial for optimal patient selection . The standard preoperative staging of patients with colorectal liver metastases includes combined abdominal CT and chest x-ray or chest CT. . It was recently demonstrated that FDG-PET as a complementary staging method improves the therapeutic management of patients with colorectal liver metastases .20 Preoperative screening with FDG-PET results in an increased survival rate of patients who undergo liver resection .14 This can be explained by the detection of occult intra - and extrahepatic metastatic disease , thus obviating futile explorations . In the current study , the sensitivity of FDG-PET/CT following neoadjuvant therapy was only 49 % compared to a sensitivity of 93.3 % in patients who did not receive neoadjuvant treatment -LRB- P < 0.0001 -RRB- . The influence of the chemotherapeutic drugs on the sensitivity of FDG-PET in detecting extrahepatic metastases is not known , but we could assume that it is influenced in a similar way . This may result in a higher-than-expected rate of nonresectable disease discovered at the time of laparotomy and more extrahepatic recurrences following resection . In our series , only three of the 48 patients -LRB- 6.25 % -RRB- who received neoadjuvant chemotherapy were found to have nonresectable disease -LRB- one had diffuse liver metastases and two had peritoneal spread -RRB- that was not discovered preoperatively by either abdominal CT or FDG-PET/CT . We believe that one of the reasons for the high operability rate is the fact that a significant number of patients underwent a baseline FDG-PET/CT before the administration of neoadjuvant chemotherapy . We therefore recommend performing a baseline FDG-PET scan for all candidates for liver resection before the administration of neoadjuvant treatment . A longer follow-up is needed to assess the results of our application of this protocol . The decreased sensitivity of FDG-PET/CT in detecting liver metastases should also be a consideration when planning the extent of liver resection . We believe that the extent of resection should be guided by additional imaging modalities , including abdominal CT and IOUS , in patients who received neoadjuvant treatment . In our series , triphasic contrast-enhanced abdominal CT had a higher sensitivity than FDG-PET/CT in detecting colorectal metastasis in patients who received neoadjuvant chemotherapy -LRB- 65.3 vs 49 % , P < 0.0001 -RRB- . The higher sensitivity of CT alone compared to FDG-PET/CT in detecting small colorectal metastasis has been reported by Ruers et al. ,20 and this may be even greater in patients who received chemotherapy . An attractive solution is the integrated PET/CT scanner on which a diagnostic triphasic abdominal CT scan can be performed at the same setting as the PET scan . There are several possible explanations for the decreased sensitivity of FDG-PET/CT in the detection of colorectal metastases following neoadjuvant therapy : Size of the lesion . The sensitivity of FDG-PET in detecting colorectal metastasis was reported as being directly related to the size of the lesions .20 We found similar results in our series -LRB- Table 4 -RRB- . The average size of the metastases following neoadjuvant treatment was significantly smaller than that in patients who did not receive chemotherapy -LRB- 33.9 mm in group 1 and 18.9 mm in group 2 , P < 0.0001 -RRB- . Two FN results in group 1 and 32 in group 2 involved tumors smaller than 1 cm . We can assume that one of the main reasons for the decreased sensitivity of FDG-PET following chemotherapy is the decrease in size of the metastases.Chemotherapy and `` metabolic shutdown . '' It has been demonstrated that the sensitivity of FDG-PET is diminished in cancer patients who undergo the examination less than 2 weeks following the administration of chemotherapy ,21 presumably due to a temporary metabolic `` shutdown . '' Although the scans in our study were done with a minimal interval of 2 weeks from the last course of chemotherapy , partial response to therapy may have caused decreased FDG uptake in metastatic lesions , making them undetectable in comparison to the physiological background uptake of FDG in the liver . This may have been a contributing factor to the FN results in our series . We found a lower sensitivity of FDG-PET -LRB- but not of CT -RRB- in detecting liver metastases following regimens including bevacizumab compared to regimens that did not include bevacizumab , although the difference did not reach statistical significance . This result may have significant clinical implications ; however , it needs to be verified in larger series.Time interval between the FDG-PET and surgery . In our series , two patients with FN results -LRB- two hepatic lesions -RRB- underwent surgery more than 2 months after FDG-PET was performed . Viable tumors were discovered at the site of the original metastases which had disappeared on FDG-PET following neoadjuvant treatment . Although it is conceivable that the relatively long interval between FDG-PET and surgery may have contributed to the FN results , we believe that these tumors may have been FN due the small size of the lesions following partial pathological response to chemotherapy.Nonavid tumors . PET avidity of the tumors can be assessed only in patients who undergo a baseline FDG-PET before neoadjuvant treatment . It has been reported that FDG-PET is less sensitive for mucinous adenocarcinoma .22 In our series , ten FN results -LRB- lesions -RRB- were in patients with nonavid mucinous adenocarcinoma -LRB- sensitivity 37.5 % -RRB- . There were two FP results in group 1 and four FP results in group 2 . Four of the six FP results were in patients who had undergone a previous hepatic resection , for which follow-up FDG-PET detected uptake in the same location of the resected metastasis . These patients underwent nonanatomic resections of the `` lesions . '' The pathological examination revealed only foreign body reaction without any tumor cells . In our current study , FDG uptake in the tumor bed following previous resection had a positive predictive value of 33 % -LRB- 2/6 -RRB- . The specificity of FDG uptake in the tumor bed for recurrence was 60 % . We believe that FDG uptake in the tumor bed following a previous liver resection is not specific for tumor recurrence , especially if the CEA levels are normal . Nguyen et al. demonstrated that FDG uptake may be high in various granulomatous lesions ,23 possibly explaining the FP results along resection margins . Therefore , biopsy or follow-up should be considered in these cases . Conclusions The sensitivity of FDG-PET in detecting colorectal hepatic metastases decreases significantly following neoadjuvant chemotherapy . This may result in a higher-than-expected rate of nonresectable disease discovered at the time of laparotomy and in more extrahepatic recurrences following resection . We recommend staging patients with a `` baseline '' contrast-enhanced FDG-PET/CT both before and after the administration of neoadjuvant therapy . The extent of hepatic resection should be guided by systematic integration of data from all additional imaging modalities -LRB- abdominal CT , IOUS -RRB- , as well as by the original imaging findings -LRB- before the neoadjuvant treatment -RRB- . We recommend resection of all metastases that achieved complete radiological response , whenever technically possible . Longer follow-up and further studies are required to justify neoadjuvant treatment and screening with FDG-PET/CT in patients with colorectal metastases to the liver who are at high risk of recurrence .

O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

colorectal liver metastases
neoadjuvant chemotherapy


Design and Implementation of an Open Source Indexing Solution for a Large Set of Radiological Reports and Images This paper hopes to share the insights we experienced during designing , building , and running an indexing solution for a large set of radiological reports and images in a production environment for more than 3 years . Several technical challenges were encountered and solved in the course of this project . One hundred four million words in 1.8 million radiological reports from 1989 to the present were indexed and became instantaneously searchable in a user-friendly fashion ; the median query duration is only 31 ms. Currently , our highly tuned index holds 332,088 unique words in four languages . The indexing system is feature-rich and language-independent and allows for making complex queries . For research and training purposes it certainly is a valuable and convenient addition to our radiology informatics toolbox . Extended use of open-source technology dramatically reduced both implementation time and cost . All software we developed related to the indexing project has been made available to the open-source community covered by an unrestricted Berkeley Software Distribution-style license . BACKGROUND Whenever a person or an institution builds a large enough collection of anything , searching through it becomes tedious or even unfeasible . An obvious solution is indexing : the process of compiling a list of pointers to items in the collection based on alphabetical order or any other item attribute . Indexing can add useful metadata , which is data about data , to a collection . A well-known example is the Science Citation Index for papers published in academic journals as envisioned by Eugene Garfield in 1955.1 Hospitals typically generate a lot of data about patients , their examinations , and their treatments . Due to the patient-oriented workflow , hospital information systems have a strong patient-centric architecture ; hence , problems tend to arise whenever information across patients is required . For purposes of business intelligence , data mining and data warehousing can provide an adequate solution . When cross-patient information is needed in a research or training context , collecting medical data will become labor-intensive or practically impossible if the relevant data only exist in a free textual form . A solution for this problem consists in building an index of all free-text words that allows for efficient queries . In our institution , the university hospital closely associated with the largest Belgian university , close to two million radiological reports and corresponding images were made instantaneously searchable by implementing a language-independent , feature-rich indexing solution . This paper hopes to share the insights we experienced during designing , building , and running an indexing system in a production environment for more than 3 years . Extended use of open-source technology dramatically reduced both implementation time and cost . METHODS To achieve a complete indexing solution that is ready for a production environment , we designed a system consisting of five major sections : the report warehouse , the search engine , specific import filters , efficient indexing , and finally the user interface . These sections correspond to five discrete phases in the implementation scheme . An overview of the architecture is presented in Figure 1 . Fig 1 . The architecture of the complete indexing system . Report Warehouse Indexing a large set of reports requires fast access to all text files to reduce the time needed to build such an index . Our radiology information system -LRB- RIS -RRB- did not comply with this rule because reports were stored in an Oracle database using a Lempel -- Ziv related compression scheme .2 To counter this problem we decided to set up an independent report warehouse that did not have an impact on RIS performance . We started out by recycling two 5-year-old servers and fitting both of them with enough hard disk capacity -LRB- 160 gigabytes -RRB- and a small uninterruptible power supply . We chose to run the open-source Free Berkeley Software Distribution -LRB- FreeBSD -RRB- operation system ,3 which had the important ability to maintain disk integrity after a crash or power outage . This technique , called soft updates , works by properly ordering file system metadata-writes to guarantee consistency and was introduced by Marshall Kirk McKusick .4 Both servers were configured as a high-availability cluster . We achieve synchronization with the RIS through two parallel processes . First , a structured query language -LRB- SQL -RRB- statement interrogates the RIS to find out which reports were cancelled since the last synchronization . Subsequently , these cancelled reports are removed from the report warehouse . The second process is similar to the first ; a list of new reports and reports updated with an addendum is obtained . Next , by means of another SQL statement , the report binary large objects are retrieved from the RIS database and decompressed . Alternatively , we could have been using a standardized , but more complex , method of capturing outbound Health Level 75 observation results unsolicited messages originating from RIS . At the time of implementation this method had the major disadvantage of only supporting converted flat text representations of the reports ; meanwhile , our RIS gained the possibility to send out rich text format -LRB- RTF -RRB- , retaining report layout and style . In an ideal situation , indexing should be built right into RIS ; this way both extracting and synchronizing reports and data become obsolete . Our report warehouse only contains validated reports by policy . Search Engine With the advent of the Internet and ever-increasing computing power , indexing software became widely available . Our report warehouse made it possible to plug in and evaluate several search engines with a limited set of reports . Due to our characteristic European setting , language independence was needed to support Dutch , French , English , and German reports . The ability to include RIS data in the index as report metadata was also an important factor ; this allows searching for -LRB- or sorting on -RRB- specific authors and requesting physicians and departments and other administrative information . Four more features we required were phrase and wildcard searching , Boolean logic , and the possibility to exclude common stop words . The section on semantics in this paper contains a detailed motivation for why these last four features are essential in a medical context . Both commercial and open-source indexing engines were tested and compared , but in-depth evaluation reports have not been included here because they have become seriously outdated after more than 3 years . Commercial engines were difficult or impossible to integrate due to their closed-source nature . Most engines could be discarded quickly because of a missing feature or a time-consuming indexing process . Note that the Google Search Appliance6 was not available in Europe at that time . We stopped looking for other candidates after we evaluated the highly recommended Swish-e indexing engine ,7 which had all the desired features and a very efficient indexing process . Import Filters Our evaluation showed that the filters provided by most search engines , including Swish-e , had issues with importing our reports . Most problems were related to file formats ; old formats were converted incorrectly , international characters often disappeared , and automatically generated text fragments -LRB- Microsoft Word AutoText -RRB- were not being filled in . One issue was observed across file formats : hyphenated words did not always make it into the index in one piece . Solving these problems required programming import filters for all historically and currently used file formats : both American Standard Code for Information Interchange flat text and flat text containing multiple international character sets , WordPerfect , Microsoft Word , and RTF . These in-house developed filters properly undo hyphenation and combine the imported report with corresponding RIS data into a single Extensible Mark-up Language -LRB- XML -RRB- file ready for indexing , an example of which is presented in Figure 2 . At our institution there is no mandatory structure for radiological reports . Additionally , we have put a quality assurance process in place ; whenever word processing software or a report template is changed , the import results are tested on international characters , automatically generated fragments , and hyphenation . Fig 2Example of a XML file containing the imported report and its corresponding RIS metadata . Efficient Indexing Indexing without tuning becomes very computing-intensive as the collection of document content , and thus the number of word/document combinations , grows . Fortunately , limiting the size of the index itself strongly decreases the computing power needed . In our situation , patient and study IDs in the radiological reports were safely excluded from the index without loss of information ; using the metadata from the RIS yielded the same results and was more efficient . A second technique consists in excluding some , but certainly not all , of the very frequent words present in almost all documents . These common stop words were found by building an index and evaluating the significance of the thousand most frequent words . Amongst the 39 manually excluded stop words , good examples like `` a , '' `` the , '' `` of , '' and closing statements like `` greetings '' were found . The very frequent word `` no '' was not excluded because its meaning was highly significant in combination with other keywords . In the section on semantics we will show that excluding common stop words leads to semantic advantages . We will also discuss building indexes in an incremental fashion in the section on scalability . User Interface Due to our completely digital and highly optimized reporting workflow ,8 the only good place to put the search interface was right at the radiologists ' fingertips . To attain this , we created a full front-end integration with RIS and the picture archiving and communication system -LRB- PACS -RRB- . By means of a generic search button , the interface can be launched from any RIS or PACS context . The integration also includes a security layer serving two distinct purposes . Firstly , our existing single-sign-on/single-sign-off mechanism allows radiologists and researchers to provide their authentication credentials using an encrypted hardware key , defying the need to type in many usernames and passwords . The second function of the security layer consists of auditing the search queries and the displayed reports . Because Internet technology was being used , it was straightforward to make a web-based interface that resembles well-known Internet search engines . Such simple and user-friendly interfaces lower the threshold for new radiologists . As shown in Figure 3 , users simply enter the desired keywords , possibly accompanied by parentheses and the binary operators `` and , '' `` or , '' and `` not . '' Wildcards are denoted by an asterisk , and quotes can be used for phrase searching . Between keywords or phrases , an implicit `` and '' is assumed . The user can quickly pick a previous query from his complete query history . The results can be sorted on a calculated page rank -LRB- default -RRB- , descending and ascending report time , and RIS metadata . Useful query feedback is shown : the number of results , the query duration , and the excluded stop words . Fast reviewing of reports is essential ; to meet this end , the user can choose between displaying summarized or full reports -- in both cases , keywords are highlighted and the matching ranking score is displayed graphically . With a single click of the mouse , radiologists can open both images in PACS and the historical reports of the selected patient . Fig 3 . The user interface for radiologists and researchers . Alas , trying to comprehend the ranking code of Swish-e is not for the faint of heart because of its inherent complexity and the speed optimizations used . The algorithms take into account the binary structure of the search query , details of keyword occurrences , and the relative frequency of words in the index . Luckily , unlike searching the Internet , queries in a radiological context tend to be very specific -LRB- see results -RRB- , making fine-tuning of ranking unnecessary . RESULTS Preceding the go-live of our indexing solution , over one million reports and associated data were extracted from the RIS . Some filter fine-tuning was necessary to pass all reports through the layer of import filters . The initial import was spread over a 14-day period without performance penalties for production systems . At the time of writing , our report warehouse contained 1.8 million reports and corresponding administrative data , their XML versions and backup copies not included . The system holds 7.2 million files in total . The oldest report dates back to 1989 , but not many reports of that era were available in digital form . Each night the index is rebuilt completely , which facilitates early detection of disc decay because all XML reports get reread in the course of this process . Alternatively , incremental reindexing could have been used . Building a full index is very efficient ; it only takes 76 min worth of CPU time on 8-year-old hardware to process 104 million words . Currently , the index contains 332,088 unique words in four languages , including names of patients and requesting physicians . Alas , all spelling mistakes ever made by our typists and radiologists are also included . The usefulness of the indexing solution is indicated by the fact that all resident radiologists make use of it despite the nonmandatory character . Figure 4 illustrates the nearly linear increase in users since the initial deployment in May 2004.9 Fig 4 . The increase in users over time . Indexing reports added useful metadata to our collection of radiological examinations . Early generations of the RIS did not track the supervisor of a radiologist in training ; instead , the supervisor 's name was added manually to the text of a report . Indexing made it possible to retrieve historical supervisor information . Having an independent report warehouse available that contains all radiological reports avoids vendor lock-in . During PACS implementation we encountered a 6-month delay in the PACS/RIS front-end integration due to a difficult multivendor situation . We successfully avoided this delay by making historical reports available at the radiologists ' work stations through the report warehouse . Query Analysis The search engine was consulted 7,071 times by our radiologists mainly to find interesting cases for research or training purposes . Occasionally , a query is performed to find a recent clinical exam in a fast and convenient manner . Table 1 presents the usage of search engine features . Apparently , 61 % of the radiologists are aware of and use at least one of the features despite the lack of formal training ; 39 % of them have never used a search engine feature . Nine in ten queries consist of keywords only , which means that results are mostly found without having to resolve to more complicated queries . Users tend to start out with a simple query and quickly skim across the clearly presented and highlighted results . Binary operators turned out to be the most popular feature , especially the `` and '' operator , which is used in 4.6 % of the queries to narrow the returned results . Phrase and wildcard searching each are used in 3 % of the queries , whereas parentheses account for 0.6 % . Table 1 . The Usage of Search Engine FeaturesFeatureQueries -LRB- n = 7,071 -RRB- Users -LRB- n = 38 -RRB- Keywords only6 ,344 -LRB- 89.7 % -RRB- 15 -LRB- 39 % -RRB- Phrases215 -LRB- 3.0 % -RRB- 14 -LRB- 37 % -RRB- Wildcards208 -LRB- 2.9 % -RRB- 14 -LRB- 37 % -RRB- Binary operator `` and '' 325 -LRB- 4.6 % -RRB- 15 -LRB- 39 % -RRB- Binary operator `` not '' 117 -LRB- 1.7 % -RRB- 9 -LRB- 24 % -RRB- Binary operator `` or '' 83 -LRB- 1.2 % -RRB- 8 -LRB- 21 % -RRB- Parentheses40 -LRB- 0.6 % -RRB- 6 -LRB- 16 % -RRB- Querying the index is very fast ; the median query duration is 31 ms. The most complex query took 2.6 s : results for each keyword need to be binary combined and ranked ; a processing delay is noticeable in the case of very frequent keywords or when broad wildcards are used . The users enter fairly specific queries ; medially , only 36 results are returned . A statistical analysis of the query duration and the number of results can be found in Table 2 . The cumulative frequency distribution of the query duration , presented in Figure 5 , can give the reader an idea of the response rate of the search engine . Caching performed by the operating system has a strong influence on the query duration because 36 % of the queries yield results within the 13-ms average access time of the disk drive used ; Figure 6 visualizes the relevant detail of the frequency distribution . Table 2 . Statistical Analysis of the Query Duration and the Number of Results Query Duration -LRB- ms -RRB- Number of ResultsMedian3136Maximum2 ,598962,629 Mean1295 ,878 Standard deviation27459 ,286 Skewness4 .415 Kurtosis24237Fig 5 . The cumulative frequency distribution of the query duration -LRB- ms -RRB- . Fig 6 . Detail of the frequency distribution of the query duration -LRB- ms -RRB- , showing the influence of caching performed by the operating system . Open Source Our indexing solution is a prime example of an open-source success story ; extended use of this technology dramatically reduced both implementation time and cost . Designing and building the system took a mere 6 weeks , which was less than the `` red tape '' time needed to approve a commercial offering . Apart from the implementation man-hours , the total cost of ownership consists of low-budget hardware and support costs . Support calls are rare due to the high-availability design of the system and the excellent stability of the Swish-e search engine . In fact , users almost never call ; the interface is very user friendly and similar to well-known interfaces of Internet search engines . Furthermore , passwords can not be forgotten because of the single-sign-on mechanism . About four times a year , manual intervention is required when the synchronization link with the RIS breaks down ; in this case , the administrator is automatically notified by e-mail . There has only been one hardware failure during the system 's lifetime , a broken power supply , but this did not cause any downtime . All software we developed in the course of this project has been made available10 to the open-source community covered by an unrestricted BSD-style license . The source code included linking and synchronization with a RIS , five import filters , the security layer and audit logging , the complete user interface , and the integration code for both RIS and PACS . DISCUSSION Although indexing technology has undoubtedly been used in a clinical setting before , the authors believe they have followed a novel approach in designing and implementing a complete and scalable indexing solution . This high-availability and high-performance system has been running without issues for more than 3 years in a production environment . Furthermore , the independent and flexible design of the platform allows for hosting other applications ; several research projects have already found a home under its wings . In this section , Internet indexing technology , scalability , and semantics are briefly discussed . Internet Indexing Technology Our decision to make use of Internet indexing technology in a medical context was straightforward because of its ubiquitous presence , the available software components , and the maturity this technology has gained over the years . Three generations of automated Internet indexing systems can be identified as to their methods of compiling their data sets , their search interfaces , and their associated etymological metaphors and mythologies .11 Firstly , the Archie search engine dating back to 1990 allowed for single keyword and regular expression searches of the file lists of anonymous File Transfer Protocol sites . Gopher , a distributed document network protocol preceding the World Wide Web ,12 featured a second generation ; the Veronica system allowed for Boolean queries of directory information and filenames . Thirdly , search engines that compile searchable databases of information accessible via the World Wide Web are in wide use today , such as the popular Google .13 Indexing technology slowly found its way into computer operating systems like Microsoft Corporation 's Windows Vista -LRB- released in 2007 -RRB- and Apple Corps ' Mac OS X 10.4 -LRB- released in 2005 -RRB- . Two aspects of Internet indexing technology could not be applied to our medical documents because of absent hyperlinking . The first was `` spidering '' : robots crawling the web to search for new and updated material by investigating hyperlinks . However , this did not pose a problem because the RIS kept track of new and updated reports . Secondly , page-ranking algorithms could not use rules based on the number of referrals by other documents . A nationwide study conducted by Vorbeck et al14 in Austria , apart from the Alps , a country similar to Belgium , showed that radiologists were already familiar with the Internet in 1999 . Today , all radiologists at our institution have a broadband Internet connection at home , including secure access to the electronic medical records and the PACS . Scalability Generally , the duration of building an index should be less than the desired timeframe in which new or updated documents show up in the system . Build times can increase beyond this point when document access speeds decrease or when the total size of the document collection grows . To counter this limitation , indexes can be built incrementally ; however , care must be taken because some indexers do not recalculate frequencies of very common words for automatic exclusion . Similarly , indexes can be built in parallel ; both mechanisms make use of index merging . Judging by our current results and the obsolete hardware used , we estimate that our indexing system scales up by two orders of magnitude , the main bottleneck being the nondistributed index . To scale a cross-patient search engine , hospital-wide privacy considerations have to be taken into account . The hospital 's policy on patient privacy should be implemented in strict access rules based on document metadata . Semantics A disadvantage of keyword-based searching is the simple fact that radiologists or researchers want to find the meaningful concepts behind their keywords . To make word indexing more or less suitable for concept searching , four search engine features are necessary . Firstly , because medical terms mostly consist of more than one word , it has to be possible to search for phrases . Page-ranking algorithms can help by giving a higher score to documents with adjacent keywords , but in practice , the sole use of this technique alone is not specific enough . Secondly , very frequent words with low significance can hamper phrase searching and should not be indexed ; when `` of '' and `` the '' are excluded , the phrases `` MRI of the brain '' and `` MRI brain '' yield the same results . A third helpful feature is Boolean logic to facilitate binary operations like `` and , '' `` or , '' and `` not '' in combination with parentheses . Finally , a word-stemming technique can be used to find all words with a common root . Although we did not implement word stemming because our multilanguage set-up required building a separate stemmed index for each language , the use of wildcards turned out to be an adequate alternative . Word stemming is part of the broader method of fuzzy indexing , which can be used to find words with similar pronunciations . To find conceptual information , users build their queries in multiple steps ; hence , the user interface should allow for fast reviewing of results . Table 3 shows a typical user refining his query . Note that structured reporting , which uses standardized information concepts , does not have the semantic problems of free-text reporting . Table 3 . Refining of a Query by a Typical UserTimeQueryNumber of Results12 : 00cirrhose52812 : 02cirrhose ascites18312 : 02cirrhose ascites MR not angio * 100 A lot of research regarding medical semantics has been conducted , ranging from natural language processing techniques15 -- 17 to building unified medical lexica18 ,19 for multiple languages .20 The authors hope to combine the indexing solution with some of these scientific methods and technology regarding the semantic web21 in the near future . CONCLUSION In this digital age , special care should be taken that the ever-growing amount of on-line information does not deteriorate into an inaccessible swamp of mere bits and bytes . On should strive to provide fast , clear , and easy access to information sources and , hence , increase the total value of stored data . Designing and implementing an indexing solution for a large set of radiological reports and images turned out to be a technically challenging but educational endeavor . For research and training purposes , it certainly is a valuable and convenient addition to our radiology informatics toolbox . The use of open-source technology is highly recommended to reduce both implementation time and cost .

O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

radiology information systems (ris)
pacs integration
medical informatics applications
radiology reporting
document indexing


e-Health Code of Ethics -LRB- May 24 -RRB- The Internet is changing how people receive health information and health care . All who use the Internet for health-related purposes must join together to create an environment of trusted relationships to assure high quality information and services ; protect privacy ; and enhance the value of the Internet for both consumers and providers of health information , products , and services . The goal of the e-Health Code of Ethics is to ensure that people worldwide can confidently and with full understanding of known risks realise the potential of the Internet in managing their own health and the health of those in their care . The final e-Health Code of Ethics , presented in this paper , has been prepared as a result of the `` e-Health Ethics Summit , '' which convened in Washington DC on 31 January 2000 - 2 February 2000 . The summit , organized by the Internet Healthcare Coalition and hosted by the World Health Organisation/Pan-American Health Organisation -LRB- WHO/PAHO -RRB- , was attended by a panel of about 50 invited experts from all over the world and produced the foundation for a draft code , which was released 18 February -LSB- 1 -RSB- for an online public consultation period which ended on 14 April 2000 . The final Washington e-Health Code of Ethics sets forth guiding principles under eight main headings : candor ; honesty ; quality ; informed consent ; privacy ; professionalism in online health care ; responsible partnering ; and accountability . Vision Statement The goal of the e-Health Code of Ethics is to ensure that people worldwide can confidently and with full understanding of known risks realise the potential of the Internet in managing their own health and the health of those in their care . Introduction The Internet is changing how people give and receive health information and health care . All people who use the Internet for health-related purposes-patients , health care professionals and administrators , researchers , those who create or sell health products or services , and other stakeholders-must join together to create a safe environment and enhance the value of the Internet for meeting health care needs . Because health information , products , and services have the potential both to improve health and to do harm , organisations and individuals that provide health information on the Internet have obligations to be trustworthy , provide high quality content , protect users ' privacy , and adhere to standards of best practices for online commerce and online professional services in health care . People who use Internet health sites and services share a responsibility to help assure the value and integrity of the health Internet by exercising judgment in using sites , products , and services , and by providing meaningful feedback about online health information , products , and services . Definitions Health information includes information for staying well , preventing and managing disease , and making other decisions related to health and health care . It includes information for making decisions about health products and health services . It may be in the form of data , text , audio , and/or video . It may involve enhancements through programming and interactivity . Health products include drugs , medical devices , and other goods used to diagnose and treat illnesses or injuries or to maintain health . Health products include both drugs and medical devices subject to regulatory approval by agencies such as the U.S. Food and Drug Administration or U.K. Medicines Control Agency and vitamin , herbal , or other nutritional supplements and other products not subject to such regulatory oversight . Health services include specific , personal medical care or advice ; management of medical records ; communication between health care providers and/or patients and health plans or insurers , or health care facilities regarding treatment decisions , claims , billing for services , etc. ; and other services provided to support health care . Health services also include listserves , bulletin boards , chat rooms , and other online venues for the exchange of health information . Like health information , health services may be in the form of data , text , audio , and/or video , and may involve enhancements through programming and interactivity . Anyone who uses the Internet for health-related reasons has a right to expect that organisations and individuals who provide health information , products or services online will uphold the following guiding principles : Guiding Principles Table 1 Guiding Principles 1 . Disclose information that if known by consumers would likely affect consumers ' understanding or use of the site or purchase or use of a product or service . CandorPeople who use the Internet for health-related purposes need to be able to judge for themselves that the sites they visit and services they use are credible and trustworthy . Sites should clearly indicate who owns or has a significant financial interest in the site or servicewhat the purpose of the site or service isFor example , whether it is solely educational , sells health products or services , or offers personal medical care or adviceany relationship -LRB- financial , professional , personal , or other -RRB- that a reasonable person would believe would likely influence his or her perception of the information , products , or services offered by the siteFor example , if the site has commercial sponsors or partners , who those sponsors/partners are and whether they provide content for the site 2 . Be truthful and not deceptive HonestyPeople who seek health information on the Internet need to know that products or services are described truthfully and that information they receive is not presented in a misleading way . Sites should be forthright in all content used to promote the sale of health products or servicesin any claims about the efficacy , performance , or benefits of products or servicesThey should clearly distinguish content intended to promote or sell a product , service , or organisation from educational or scientific content . 3 . Provide health information that is accurate , easy to understand , and up to date . QualityTo make wise decisions about their health care , people need and have the right to expect that sites will provide accurate , well-supported information and products and services of high quality . To assure that the health information they provide is accurate , e-Health sites and services should make good faith efforts to evaluate information rigorously and fairly , including information used to describe products or servicesprovide information that is consistent with the best available evidenceassure that when personalized medical care or advice is provided that care or advice is given by a qualified practitionerindicate clearly whether information is based on scientific studies , expert consensus , or professional or personal experience or opinionacknowledge that some issues are controversial and when that is the case make good faith efforts to present all reasonable sides in a fair and balanced wayFor example , advise users that there are alternative treatments for a particular health condition , such as surgery or radiation for prostate cancerInformation and services must be easy for consumers to understand and use . Sites should present information and describe products or services in language that is clear , easy to read , and appropriate for intended usersFor example , in culturally appropriate ways in the primary language -LRB- or languages -RRB- of the site 's expected audiencein a way that accommodates special needs users may haveFor example , in large type or through audio channels for users whose vision is impairedSites that provide information primarily for educational or scientific purposes should guarantee the independence of their editorial policy and practices by assuring that only the site 's content editors determine editorial content and have the authority to reject advertising that they believe is inappropriate . Consumers have a right to expect that the information they receive is up to date . Sites should clearly indicate when the site published the information it provides -LRB- and what version of the information users are seeing if it has been revised since it was first published -RRB- when the site most recently reviewed the informationwhether the site has made substantive changes in the information and if so , when the information was most recently updated and Provide the information users need to make their own judgments about the health information , products , or services provided by the site . Individuals need to be able to judge for themselves the quality of the health information they find on the Internet . Sites should describe clearly and accurately how content is developed for the site by telling userswhat sources the site or content provider has used , with references or links to those sourceshow the site evaluates content and what criteria are used to evaluate content , including on what basis the site decides to provide specific links to other sites or servicesFor example , by describing the site 's editorial board and policiesWhen health products or services are subject to government regulation , sites should tell users whether those products -LRB- such as drugs or medical devices -RRB- have been approved by appropriate regulatory agencies , such as the U.S. Food and Drug Administration or U.K. Medicines Control Agency 4 . Respect users ' right to determine whether or how their personal data may be collected , used , or shared . Informed ConsentPeople who use the Internet for health-related reasons have the right to be informed that personal data may be gathered , and to choose whether they will allow their personal data to be collected and whether they will allow it to be used or shared . And they have a right to be able to choose , consent , and control when and how they actively engage in a commercial relationship . Sites should clearly disclose that there are potential risks to users ' privacy on the InternetFor example , that other organisations or individuals may be able to collect personal data when someone visits a site , without that site 's knowledge ; or that some jurisdictions -LRB- such as the European Union -RRB- protect privacy more stringently than othersSites should not collect , use , or share personal data without the user 's specific affirmative consent . To assure that users understand and make informed decisions about providing personal data , sites should indicate clearly and accurately what data is being collected when users visit the siteFor example , data about which parts of the site the user visited , or the user 's name and email address , or specific data about the user 's health or online purchaseswho is collecting that dataFor example , the site itself , or a third partyhow the site will use that dataFor example , to help the site provide better services to users , as part of a scientific study , or to provide personalised medical care or advicewhether the site knowingly shares data with other organisations or individuals and if so , what data it shareswhich organisations or individuals the site shares data with and how it expects its affiliates to use that dataFor example , whether the site will share users ' personal data with other organisations or individuals and for what purposes , and note when personal data will be shared with organizations or individuals in other countriesobtain users affirmative consent to collect , use , or share personal data in the ways describedFor example , to collect and use the visitor 's personal data in scientific research , or for commercial reasons such as sending information about new products or services to the user , or to share his or her personal data with other organisations or individualswhat consequences there may be when a visitor refuses to give personal dataFor example , that the site may not be able to tailor the information it provides to the visitor 's particular needs , or that the visitor may not have access to all areas of the site `` E-commerce '' sites have an obligation to make clear to users when they are about to engage in a commercial transaction and to obtain users ' specific affirmative consent to participate in that commercial transaction . 5 . Respect the obligation to protect users ' privacy . PrivacyPeople who use the Internet for health-related reasons have the right to expect that personal data they provide will be kept confidential . Personal health data in particular may be very sensitive , and the consequences of inappropriate disclosure can be grave . To protect users , sites that collect personal data should take reasonable steps to prevent unauthorised access to or use of personal dataFor example , by `` encrypting '' data , protecting files with passwords , or using appropriate security software for all transactions involving users ' personal medical or financial datamake it easy for users to review personal data they have given and to update it or correct it when appropriateadopt reasonable mechanisms to trace how personal data is usedFor example , by using `` audit trails '' that show who viewed the data and whentell how the site stores users ' personal data and for how long it stores that dataassure that when personal data is `` de-identified '' -LRB- that is , when the user 's name , email address , or other data that might identify him or her has been removed from the file -RRB- it can not be linked back to the user 6 . Respect fundamental ethical obligations to patients and clients . Professionalism in Online Health CarePhysicians , nurses , pharmacists , therapists , and all other health care professionals who provide specific , personal medical care or advice online should abide by the ethical codes that govern their professions as practitioners in face-to-face relationshipsdo no harmput patients ' and clients ' interests firstprotect patients ' confidentialityclearly disclose any sponsorships , financial incentives , or other information that would likely affect the patient 's or client 's perception of professional 's role or the services offeredclearly disclose what fees , if any , will be charged for the online consultation and how payment for services is to be madeobey the laws and regulations of relevant jurisdiction -LRB- s -RRB- , including applicable laws governing professional licensing and prescribing and Inform and educate patients and clients about the limitations of online health care . The Internet can be a powerful tool for helping to meet patients ' health care needs , but users need to understand that it also has limitations . Health care professionals who practice on the Internet should clearly and accurately identify themselves and tell patients or clients where they practice and what their professional credentials aredescribe the terms and conditions of the particular online interactionFor example , whether the health care professional will provide general advice about a particular health condition or will make specific recommendations and or referrals for the patient or client , or whether the health care professional can and will or can not and will not prescribe medications in the particular situationmake good faith efforts to understand the patient 's or client 's particular circumstances and to help him or her identify health care resources that are available locallyFor example , to help the patient or client determine whether particular treatment is available in his or her home community or only from providers outside his or her communitygive clear instructions for follow-up care when appropriate or necessaryHealth care professionals who offer personal medical services or advice online should clearly and accurately describe the constraints of online diagnosis and treatment recommendationsFor example , providers should stress that because the online health care professional can not examine the patient , it is important for patients to describe their health care needs as clearly they canhelp `` e-patients '' understand when online consultation can and when it can not and should not take the place of a face-to-face interaction with a health care provider 7 . Ensure that organisations and sites with which they affiliate are trustworthy . Responsible PartneringPeople need to be confident that organisations and individuals who operate on the Internet undertake to partner only with trustworthy individuals or organisations . Whether they are for-profit or nonprofit , sites should make reasonable efforts to ensure that sponsors , partners , or other affiliates abide by applicable law and uphold the same ethical standards as the sites themselvesinsist that current or prospective sponsors not influence the way search results are displayed for specific information on key words or topicsAnd they should indicate clearly to users whether links to other sites are provided for information only or are endorsements of those other siteswhen they are leaving the siteFor example , by use of transition screens 8 . Provide meaningful opportunity for users to give feedback to the site . AccountabilityPeople need to be confident that organisations and individuals that provide health information , products , or services on the Internet take users ' concerns seriously and that sites make good faith efforts to ensure that their practices are ethically sound . e-Health sites should indicate clearly to users how they can contact the owner of the site or service and/or the party responsible for managing the site or serviceFor example , how to contact specific manager -LRB- s -RRB- or customer service representatives with authority to address problemsprovide easy-to-use tools for visitors to give feedback about the site and the quality of its information , products , or servicesreview complaints from users promptly and respond in a timely and appropriate mannerSites should encourage users to notify the site 's manager -LRB- s -RRB- or customer service representatives if they believe that a site 's commercial or noncommercial partners or affiliates , including sites to which links are provided , may violate law or ethical principles . and Monitor their compliance with the e-Health Code of Ethics . e-Health sites should describe their policies for self-monitoring clearly for users , and should encourage creative problem solving among site staff and affiliates .

O O O B O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O