with ever - improving imaging technologies and high - performance 
 computational power , the complexity and scale of brain imaging 
 data have continued to grow at an explosive pace . recent advances 
 in imaging technologies , such as molecular imaging using positron 
 emission tomography ( pet ) , structural imaging using 
 high - resolution magnetic resonance ( mr ) , as well as quantitative 
 electrophysiological cortical mapping using electroencephalography 
 ( eeg ) , have allowed an increased understanding of normal and 
 abnormal brain structures and functions [ 14 ] it is well 
 understood that normal brain function is dependent on the 
 interactions between specialized regions of the brain which 
 process information within local and global networks . consequently , there is a need to integrate the acquired 
 multimodality data in order to obtain a more detailed 
 understanding about process interaction in a complex biological 
 system . in addition to integration , advanced computational tools 
 need to be developed which allow quantitative analysis of a 
 variety of functional patterns as well as the design of a software 
 environment that allows quantitative assessment of relationships 
 between diverse functional and anatomical features . although this 
 is an area of active research , current state - of - the - art 
 technologies are still suboptimal with respect to multimodality 
 integration and quantitative assessment of qualitatively distinct 
 neuroimaging datasets in patients with structurally abnormal 
 brains . this paper presents a multimodality database for 
 neuroimaging , electrophysiological , as well as clinical data 
 specifically designed for the assessment of epileptic foci in 
 patients with epilepsy , and can serve as a standard template for 
 other diagnostic procedures which employ multiple imaging 
 technologies . we 
 discuss previous work by other investigators and how it relates to 
 the proposed framework . in section 3 , we introduce a 
 novel image analysis method able to assess quantitatively the 
 relationship between imaging and electrophysiological data . in 
 section 4 , we describe the integration of diverse 
 data sets into an extendable database structure and describe 
 available queries for quantitative analysis of multimodality data . section 5 details the implementation issues and shows 
 the applications of the presented environment . finally , in 
 section 6 we discuss the strength and limitations of 
 the proposed system and conclude with potential future directions . the accurate segmentation of anatomical brain structures from 
 medical images and their compact geometrical representation in a 
 rigorous computational framework is difficult due to the 
 complexity and physiological variability of the structures under 
 study . in the past , a large number of techniques have been applied 
 to achieve spatial standardization of the cortical surface within 
 a group of subjects , but only a few methods are currently 
 available for the definition of homotopic volume elements within 
 the whole brain . for example , thompson et al . , lohmann 
 , and vaillant et al .   used a manually 
 labeled atlas brain , which was then warped to fit an individual 
 subject 's brain surface with subsequent transfer of labels onto 
 the subject 's cortical surface . another related method is the 3d 
 stereotactic surface projection ( 3d - ssp ) technique , which 
 is based on the extraction of subcortical functional data onto a 
 predefined set of surface pixels following stereotactic anatomical 
 standardization of an individual brain . more recently , conformal 
 mapping techniques have gained a wider application in brain 
 mapping . for example , hurdal and stephenson 
 proposed a discrete mapping approach that uses spherical packing 
 in order to produce a  flattened  image of the cortical surface 
 onto a sphere yielding maps that are quasiconformal approximations 
 to classical conformal maps . moreover , gu et al . 
 proposed optimization of the conformal parameterization method by 
 composing an optimal mbius transformation so that it 
 minimizes the landmark mismatch energy . based on the work of the 
 above investigators as well as on our previous work with 
 landmark - constrained conformal surface mapping , we have 
 developed a landmark - constrained conformal mapping of the brain 
 where the mapping accuracy is achieved through matching of various 
 cortical landmarks ( e.g. , central sulcus , sylvian fissure , 
 interhemispheric fissure ) . these landmarks are manually defined in 
 the subject 's native space and subsequently aligned in the 
 spherical domain using landmark - constrained conformal mapping . finite cortical elements are defined geometrically in the 
 spherical domain and subsequently reversely mapped into the 
 subject 's native space where all data analyses take place . as the 
 position of cortical landmarks is affected by structural 
 abnormalities , this method incorporates both physiological 
 variations as well as structural abnormalities into the mapping 
 process . the creation of brain image databases which allow storage , 
 organization , and the sharing of processed brain data between 
 investigators with different scientific backgrounds in easily 
 accessible archives is one of the main goals of the newly emerging 
 field of neuroinformatics . this objective is becoming increasingly 
 important for researchers in the brain imaging field as an 
 unprecedented amount of brain data is now acquired using 
 complementing modalities which need to be analyzed based on an 
 overarching strategy [ 1315 ] . following the model of other 
 scientific fields which were revolutionized through the 
 implementation of powerful database structures such as the genbank 
 ( http://www.ncbi.nlm.nih.gov ) for the field of genomics or the protein data bank ( http://www.rcsb.org/pdb ) for the field of 
 proteomics , several neuroimaging databases were generated in the 
 past decade in order to fully mine the information present in the 
 acquired data . the visible human project website http://www.nlm.nih.gov/research/visible/visible_human.html 
 researchers to view high - resolution image sections of two human 
 cadavers and is widely used in education and training . more 
 comprehensive human brain databases which attempt to combine the 
 expertise of computer programmers , statisticians , and basic 
 researchers in an attempt to develop neuroinformatics tools for 
 interdisciplinary collaboration are the icbm database 
 http://www.loni.ucla.edu 
 which includes pet , mri , fmri , eeg , and 
 meg modalities   as well as the brainmapdbj 
 http://www.brainmapdbj.org 
 and the echbd databases which both integrate pet and fmri image 
 data . in comparison to these multiinstitutional 
 large - scope archives , our database is specific to the task of 
 understanding the mechanisms of epileptogenesis in patients with 
 epilepsy who are evaluated for resective surgery . whereas the 
 above - mentioned archives integrate the data within the concept of 
 a probabilistic atlas of the human brain , a major consideration of 
 our database design was the analysis and integration of data in 
 native space . as a consequence , our approach not only allows the 
 integration of complementing modalities ( pet , mri , eeg ) in native 
 space of each individual patient , but also provides software tools 
 able to quantify the relationship between these diverse 
 modalities . consequently , the developed framework has the 
 potential to generate quantitative measures which characterize the 
 state of the brain in more detail than would be possible with each 
 individual modality . the main objective of presurgical evaluation of 
 patients with medically refractory epilepsy is to define the 
 boundaries of epileptogenic brain regions to be resected . towards 
 this goal , definition of the epileptogenic cortex by intracranial 
 subdural eeg recording however , the 
 accuracy of foci localization using subdural electrodes depends 
 greatly on location of electrodes placed on the brain surface , and 
 selection bias ( i.e. , area of cortex sampled ) is a major 
 limitation . in order to guide the placement of subdural 
 electrodes , a combination of noninvasive anatomical and functional 
 imaging , such as mr and pet imaging , is frequently used . these 
 modalities provide the epilepsy surgery team with important 
 information to guide placement of subdural electrodes over 
 epileptogenic brain regions based on the position and extent of 
 anatomical and functional abnormalities [ 1820 ] as well as 
 the seizure semiology ( i.e. , symptoms of seizures ) . furthermore , 
 after the intracranial eeg electrodes are implanted , each 
 electrode records a number of electrophysiological parameters that 
 need to be related to the imaging information in order to decide 
 on the most effective course of surgical intervention . to this 
 end , we designed an extendable multimodality database which allows 
 assessment of relationships among various modalities across a 
 large patient population . a focal point of our design was the 
 extensibility of the database scheme in order to allow easy 
 inclusion of data originating from newly developed pet tracers or 
 the inclusion of emerging modalities such as diffusion tensor 
 imaging ( to incorporate a measure of brain connectivity ) or 
 susceptibility weighted imaging ( to assess venous vasculature in 
 brain tissue ) . pet imaging using the tracers [ f-18]deoxy - l - glucose ( fdg ) and [ c-11]flumazenil ( fmz ) was performed as part of the clinical management of patients undergoing evaluations for 
 epilepsy surgery . pet studies were performed using the cti / siemens 
 exact / hr scanner ( knoxville , tenn , usa ) with a reconstructed image 
 resolution of about 5 mm fwhm . all subjects was fasted for 4 
 hours prior to the pet procedure and eeg was monitored using scalp 
 electrodes during the whole study duration . static pet images were 
 obtained based on coincidence data acquired between 
 4060 minutes post injection for the fdg and between 
 2040 minutes post injection for the fmz tracer . mri studies were performed on a ge 1.5 tesla signa unit 
 ( milwaukee , wis , usa ) . high - resolution t1-weighted images were 
 performed using a fast spoiled gradient echo ( spgr ) sequence . the 
 spgr technique generates 124 contiguous 1.5 mm sections of the 
 entire head using a 4.6/1.3/450 ( tr / te / ti ) pulse sequence , flip 
 angle of 12 degrees , matrix size of 256  256 , and fov of 
 240 mm . following coregistration of the pet image volumes to 
 the high - resolution mr image volume , all extracerebral structures 
 were removed from the mr image volume using an in - house developed 
 software package . following initial thresholding , the 
 software performs multiple erosion and dilation operations in 
 order to delete connections between the brain and the skull in 
 subdural space . finally , a connected component analysis is 
 performed and all but the largest component ( brain ) is deleted , 
 resulting in an mr image volume where all extracerebral structures 
 are removed and cortical sulci and gyri become visible . the 
 standard set of cortical landmarks consisted of the central 
 sulcus , sylvian fissure , and parieto - occipital sulcus , which were 
 subsequently used to define the spatial extent of the four brain 
 lobes ( frontal , parietal , temporal , and occipital ) . in order to 
 find the border of the occipital cortex on the lateral cortical 
 surface , both the parieto - occipital sulcus as well as the medial 
 border between the cerebellum and the occipital lobe were defined 
 on the midplane and then projected normal to the midplane onto the 
 lateral surface ( figure 1 ) . we have recently developed a landmark - constrained conformal 
 surface mapping technique [ 12 , 21 ] which allows accurate and reproducible transformation of each patient 's cortical surface to a canonical spherical domain ( the conformal brain model ( cbm ) , see 
 the appendix ) . once the cortical surface of a subject is mapped to the cbm , finite surface elements can be defined geometrically on the spherical surface at various resolution levels ( 8 , 32 , 128 , 512 , and 2048 ) 
 ( figure 2 ) . the set of surface elements can be subsequently reversely mapped 
 into native space where they represent homotopic surface elements 
 in brains of individual patients . these surface elements scale 
 proportionally to the size and shape of individual brains . moreover , in order to integrate pet and mri data , a normal fusion 
 approach is applied in native space of each subject . the extracted 
 mr brain surface is smoothed using a  roller - ball algorithm  
 based on alpha - shapes   where the parameter alpha is the 
 radius of a ball rolled over the surface . smoothing of the 
 cortical surface is achieved as all points deep inside cortical 
 sulci are recovered which can not be reached by the ball . following 
 triangulation of the brain surface , the normal vector to the 
 surface is calculated in each surface voxel . by averaging the pet 
 tracer concentration along the inverse normal vector in the 
 coregistered pet image volume , the average pet tracer 
 concentration within a 10 mm cortical mantel is calculated . by comparing the pet tracer concentration in these cortical 
 elements between a group of normal control subjects and an 
 individual patient , functionally abnormal increases or decreases 
 of pet tracer concentration can be determined in finite elements 
 of the cortical mantel . decision about the functional abnormality 
 of a cortical element of a patient at a given location can be then 
 made based on a severity index calculated as 
 ( 1)severityi=(mii)i , 

 where mi is the mean tracer 
 concentration at cortical location i of a patient 
 and i and i are 
 the mean tracer concentration and standard deviation ( sd ) at 
 cortical location i derived from a control group . values of the 
 severity index within 2 sd are assumed to represent 
 normal cortex . the severity of detected abnormalities ( outside 
 2 sd ) is then color - coded and mapped onto the 
 cortical surface allowing assessment of functional abnormalities 
 relative to anatomical cortical landmarks . our approach attempts to merge the advantages of voxel - based ( such 
 as spm ) and region - of - interest - based strategies , but at the same 
 time tries to avoid the pitfalls associated with either of these 
 methods . the semiautomated geometric parcellation procedure 
 creates cortical elements that are much larger than individual 
 voxels of the image volume , but circumvents the time consuming and 
 subjective definition of large surface - based regions of interest . moreover , as this method does not rely on asymmetry measures 
 between homotopic cortical volume elements , it is well 
 suited to detect bilateral cortical abnormalities . finally , the 
 most prominent advantage of our method is the fact that all data 
 samplings and analyses are performed in the subject 's native 
 space . by avoiding spatial warping , we allow application of this 
 method to brains that are very different from the normal adult 
 brain , such as brains of patients with tuberous sclerosis ( with a 
 large number of tubers deep inside the brain ) or children during 
 various developmental stages . all patients with intractable focal epilepsy included in the 
 present study underwent chronic eeg monitoring with subdural 
 electrode grids as part of their presurgical evaluation . subdural 
 electrode placement was guided by seizure semiology , scalp eeg 
 recordings , and cortical glucose metabolism abnormalities on pet . 
 during the chronic subdural eeg monitoring , more than two habitual 
 seizures were captured and analyzed using stellate 's sensa 5.0 
 software , yielding for each grid electrode the mean 
 interictal spike frequency as well as the normalized interictal 
 spike frequency ( normalized to the electrode with highest spike 
 frequency ) . identification of electrodes involved in seizure onset 
 and seizure spread of habitual clinical seizures was determined by 
 chronic subdural eeg monitoring ( see , e.g. , ) . in the past , we have implemented a method that allows the 
 localization of subdural grid electrodes on the cortical surface 
 [ 25 , 26 ] . in short , this method relies on the accurate 
 alignment of a lateral planar x - ray image ( figure 3(b ) ) 
 of the patient 's head with the electrode grid in place with a 
 three - dimensional surface rendering of the head 
 ( figure 3(a ) ) . upon completion , a surface view of the 
 cortex is created which corresponds to the planar x - ray image and 
 where the spatial coordinates of the four corner grid electrodes 
 of each rectangular eeg grid can be determined on the brain 
 surface ( figure 3(c ) ) . the accuracy of this method was 
 reported to be 1.24  0.66 mm with a maximal 
 misregistration of 2.7 mm . our previous results indicated that intracranial subdural 
 electrodes showing electrophysiological ictal ( seizure onset and 
 spread ) or interictal ( spike frequency ) abnormalities are either 
 overlapping or in close proximity to functional abnormalities 
 measured with pet imaging . because classical receiver 
 operator characteristic ( roc ) analysis is suboptimal in describing 
 spatially related measures , we have previously developed a spatial 
 proximity index ( spi ) . this index yields an overall 
 quantitative measure of the spatial relationship between 
 electrophysiological and functional pet image data and is 
 calculated based on the position of intracranial grid electrodes 
 relative to abnormal pet tracer concentration in cortical areas 
 ( figure 4 ) . the spi is a continuous variable that 
 equals zero for perfect overlap between seizure onset electrodes 
 and pet - defined abnormal cortical elements and increases in value 
 proportional to the distance between eeg - defined onset electrodes 
 and pet - defined abnormal cortical elements . the subdural eeg electrode array defines electrodes that are 
 either eeg positive ( e+ ) or negative ( e ) for seizure onset . moreover , electrodes located within the pet abnormality are designated as pet positive ( p+ ) and those outside the pet abnormality are designated as pet negative ( p ) . using these definitions , the ( unitless ) spatial proximity index ( spi ) is 
 computed as the ratio of the penalized total weighted distance between eeg positive and pet positive electrodes and the total number of seizure onset electrodes , 
 ( 2)spi=i=1mwidi(ei+|pi+)+i=1kpi+(ei+ )  i=1m(ei+ )  , 

 where m is the number of eeg positive electrodes , 
 k is the number of pet positive electrodes , and 
 wi is a weighting factor which 
 accounts for the varying interictal spike frequency at different 
 electrode locations . the weighting factor is either 1 or 0 for 
 both seizure onset and spread , whereas for spike frequency the 
 weighting factor is derived as the quotient between the spike 
 frequency at a particular electrode location and the maximal spike 
 frequency in the whole brain . in ( 2 ) , the first term in 
 the numerator represents the total weighted distance between all 
 seizure onset electrodes and the nearest pet positive electrode , 
 whereas the second term represents the number of all false 
 positive pet electrodes ( i.e. , pet positive electrodes which are 
 not eeg positive ) . finally the denominator reflects the total 
 number of eeg positive electrodes . because subdural electrodes are 
 always arranged in a rectangular lattice , the 
  city - block it is important to note that 
 all characteristics of the euclidian metric are preserved in the 
 city - block metric . in this metric , the distance between two 
 adjacent electrodes is 1 and the distance between two diagonal 
 electrodes is 2 . spi values were calculated separately for seizure 
 onset ( spionset ) , seizure spread 
 ( spispread ) , and interictal spike 
 frequency ( spispike ) . in order to demonstrate the computation of spi values , 
 figure 4 shows a ( 5  4 ) electrode grid with 
 seizure onset electrodes ( red ) and electrodes which are not 
 seizure onset electrodes ( yellow ) . in figure 4(a ) , the 
 pet abnormality is adjacent to the seizure onset electrodes 
 resulting in an spionset value of 2.66 , while in 
 figure 4(b ) the pet abnormality is remote to the seizure 
 onset electrodes resulting in an spionset value 
 of 5.66 . in this example , the lower spi value indicates a closer 
 spatial proximity between ( complementing ) pet and eeg modalities 
 and indicates that low spi values are associated with situations 
 in which pet imaging is successfully guiding the placement of an 
 intracranial eeg grid ( despite being nonoverlapping ) . conceptually , the database is structured in 
 three hierarchical levels : a patient level , a cluster level , and a 
 cortical element level . at the highest level ( patient level ) , clinical variables are stored such as the age of the patient , age 
 at first seizure , seizure type , seizure severity , and seizure 
 duration . an index of seizure severity was derived as the product 
 of average monthly seizure number and average seizure duration . as 
 the cortical distribution of each pet tracer ( e.g. , fdg , fmz , 
 etc . ) is distinct , the database includes separate row for 
 different pet tracers ( exam_type in pet_exam table ) . the 
 lowest level is the cortical element level , which provides 
 information with regard to the location of functional data in 
 cortical elements . cortical elements at different resolution 
 levels are represented by the coordinates of the three vertices 
 building the corners of a surface triangle ( corner_a , 
 corner_b , corner_c in the cortical_element table ) . in 
 addition , the pet table stores for each cortical element the 
 severity ( severity_index ) , and the spi table stores spi values 
 for seizure onset , seizure spread , and interictal spiking for the 
 whole brain at each resolution level . moreover , the surface 
 coordinates of each eeg grid electrode are stored at this lowest 
 level in the eeg_grid table . this table stores for each of the 
 potentially multiple electrode grids placed onto the cortex the 
 location of the corner electrodes ( corner_i - corner_iv in the 
 eeg_grid table ) . the location of all other electrodes within 
 the eeg grid can be then computed from the location of the corner 
 electrodes given a 10 mm spacing between individual 
 electrodes . electrophysiological parameter such as the seizure 
 category ( onset , spread , spiking ) , interictal spike frequency , or 
 normalized interictal spike frequency can be then mapped to a 
 particular grid electrode . given the location of each electrode , 
 the software then automatically determines the corresponding 
 cortical element at various resolution levels . finally the 
 intermediate cluster level ( cluster table ) allows the grouping of 
 cortical volume elements either into anatomical territories such 
 as the prefrontal , motor - frontal , parietal , temporal , and 
 occipital cortices , or into functional clusters 
 according to pet or eeg data . to accommodate the flexibility of an 
 evolving database schema , the database allows graceful inclusion of additional 
 modalities at each of the three levels . the xml - based approach 
 does not only accommodate the incorporation of additional data 
 structures into the database , but ensures also the scalibility of 
 the system . this was achieved through following design criteria : 
 ( 1 ) an efficient mapping algorithm that generates corresponding 
 relational schemas from xml schemas in linear time , ( 2 ) 
 the implementation of two efficient linear data mapping algorithms 
 in order to store xml data in the database , and ( 3 ) the 
 implementation of a linear xml subtree reconstruction algorithm 
 able to reconstruct the xml subtree from the database . the database was initially populated with control 
 data determined in control subjects in order to establish a 
 normative pattern . the control group consisted of 15 young adult 
 controls ( mean age 27.6  4.5 years ) who were not taking 
 any medication , and had no history of neurological or psychiatric 
 disorder . the purpose of the database is to provide an integrated framework 
 able to present relationships between functional and 
 electrophysiological parameters in an organized and , for the user , 
 easy - to - navigate fashion . a focus of our design was to merge an 
 efficient database structure with a display module , so that users 
 can grasp the results of their queries in an intuitive way . figure 6 shows the main control panel allowing access 
 to all data structures within the database and the link to an 
 integrated 3d image display module . for example , the user can request to 
  highlight  all eeg grid electrodes with spike frequency greater 
 than 20% of the maximal spike frequency and to calculate the 
 corresponding spi values . this is highly relevant in light of our 
 previous reports   indicating that in patients with 
 nonlesional neocortical epilepsy , seizures arose from areas 
 adjacent to pet - defined abnormalities , rather than from within the 
 pet - defined 
 abnormalities . in essence , this query retrieves the 
 electrophysiological data from table electrode and the pet data 
 from table cluster , respectively , based on the input patient 
 id($pid ) . after the location of an electrode with ( normalized ) 
 spike frequencies > 0.20 is determined , data representing the 
 current electrode is integrated with data characterizing the 
 distribution of abnormal pet clusters using the  join true  only in the case when the current 
 electrode is adjacent to an abnormal cortical region . in addition , 
 our design supports advanced graphical structure mining techniques 
 in order to discover nontrivial associations between multiple 
 electrophysiological parameters and their spatial relationship to 
 the pet - defined abnormalities . one of these data mining 
 techniques , the  association rule mining ,  is used to determine 
 whether certain clinical patterns result in larger spi values or 
 whether different pet tracers produce larger spi values given the 
 same clinical pattern . this information is important in order to 
 determine the clinical performance of newly developed pet tracers . two measurements that are used in the association mining process 
 are support and confidence defined as follows : 
 support(condition c ) = # of patients satisfying c / total # of patients , confidence(condition c1 , condition c2 ) = support(c1 & c2)/support(c1 ) 
 the association rule mining procedure returns all rules in the 
 form of condition 1    condition 2 , with a support and 
 a confidence greater than some user - specified thresholds . the two 
 measurements in conjunction quantify the association strength 
 between condition 1 and condition 2 . one example of an association 
 rule is 

 ( 3 ) 
 ( # of patients ( seizureduration > 3 .and . this indicates   that if a patient has had seizures for more than 3 
 years and the severity score of these seizures was on average 
 higher than 2 , then there is greater than 80% likelihood that the 
 patient 's spionset will be greater 
 than 1.5 . support(condition c ) = # of patients satisfying c / total # of patients , confidence(condition c1 , condition c2 ) = support(c1 & c2)/support(c1 ) we have utilized vtk / opengl for rendering , oracle 9i for database 
 applications , and c / c++ for implementation of computationally 
 intensive algorithms . furthermore , we separated the computational 
 components from other functionality and created a standalone 
 programming library consisting of the following modules : ( 1 ) 
 conformal mapping ; ( 2 ) objective definition of pet abnormalities ; 
 ( 3 ) eeg grid overlay ; ( 4 ) calculation of spi values ; ( 5 ) 
 extendable database component ; and ( 6 ) graphical output 
 module . with the expected growth of knowledge about the evolution of 
 epileptic foci as well as the emergence of new computational 
 methods that require diverse information inputs , it is impractical 
 to design a fixed database schema . as new requirements arise , it 
 is critical that the database schema adjusts gracefully to these 
 new requirements . in contrast to traditional database systems , 
 where evolution of the database schema to new requirements is 
 problematic for dynamic applications , we applied a flexible data 
 model based on xml ( extensible markup language ) [ 29 , 31 , 32 ] 
 due to its extensibility and flexibility nature . while in the past 
 we have successfully developed an xml storage and query system , in 
 which the database schema was automatically created from an xml 
 schema , this technique needed to be extended to include 
 support for the processing and storage of spatial information . twelve children ( mean age 5.2  4.3 years , age range 1 to 
 14.8 years ) with medically intractable partial epilepsy were 
 analyzed . all children were diagnosed with unilateral seizure foci 
 based on seizure semiology , scalp ictal , and intracranial eeg as 
 well as fdg pet , which were performed as part of their presurgical 
 evaluation . the region of epileptic focus as verified by eeg 
 showed hypometabolism in all cases . no cortical or subcortical 
 lesions on mri scans were observed ; however , patients with pure 
 hippocampal atrophy were included . all studies were performed in 
 accordance with guidelines stipulated by the ethics committee of 
 wayne state university . our application provides integration of anatomical , 
 functional , and electrophysiological data as shown in 
 figure 7 . upon retrieval from the database , the 
 software displays a surface rendering of the selected patient 's 
 brain with functional data derived from pet imaging mapped 
 directly onto the cortical surface . in addition , each eeg 
 electrode is color coded according to its electrophysiological 
 characteristics ( red = seizure onset , yellow = seizure spread , 
 green = normal ) and is displayed directly on the cortical surface . as the display is fully three - dimensional , the user can 
 interactively rotate and zoom the brain in order to inspect the 
 spatial relationship between eeg electrodes and pet abnormalities 
 from various viewing angles . in addition to the multimodality display , our software also allows 
 calculation of spi values in order to allow quantitative 
 comparison between localizing information based on clinical 
 readings ( onset / spread ) and localizing information obtained based 
 on a semiquantitative analysis of interictal spike frequency . figure 8 shows images determined in a patient studied 
 with both fdg as well as fmz pet , and reports spi values which 
 were calculated with regard to seizure onset electrodes and 
 electrodes with high interictal spike frequency ( > 40% of 
 maximal spike frequency ) . the user is able to select various 
 thresholds of spike frequency and interactively assess the spatial 
 relationship between frequently spiking cortical tissue , the 
 location of seizure onset , and areas of functionally abnormal 
 cortex . the spatial relationship between these areas is then 
 encoded in spi values and stored for further statistical 
 assessment . usually , eeg abnormalities are located adjacent to extensive pet 
 abnormalities and might be reversible after surgical intervention . in addition , remote areas of pet abnormality might exist , which 
 possibly indicate secondary epileptic foci initially triggered by 
 the primary focus but which might mature with time and become 
 independent . this phenomenon is at present time poorly understood 
 and possible mechanisms are discussed elsewhere . there 
 are strong indications , however , that the relationship between 
 electrophysiology and molecular function is complimentary and that 
 the eeg and pet modalities characterize different aspects of brain 
 tissue epileptogenicity . with ever - improving imaging technologies and boost 
 in computational power , the medical imaging field has experienced 
 a tremendous increase in the amount of information collected . although medical imaging modalities provide diverse quantitative 
 and qualitative information , each modality has its distinct 
 strengths and limitations . while t1-weighted mr images yield 
 high - resolution anatomical images , the obtained functional 
 information is limited . in contrast , molecular imaging using a 
 variety of pet tracers provides accurate functional information , 
 unfortunately with significantly less anatomical detail as 
 compared to mr imaging . nevertheless , both modalities provide 
 invaluable clinical information to the physicians even when 
 utilized qualitatively . in the past , our group has successfully 
 used data from pet , mr and eeg in the presurgical evaluation of 
 epileptic children and showed a significant improvement in 
 surgical outcome when using such multimodality approach 
 [ 33 , 34 ] . it appears that further improvement in the 
 localization of epileptogenic brain tissue might be derived from a 
 tighter integration of all presurgical data ( anatomical , 
 functional , electrophysiological , and clinical ) within a 
 comprehensive database . once data from multiple imaging and 
 nonimaging modalities are combined within a rigorous computational 
 framework , the information entailed in one modality may be used to 
 enhance or reinterpret information derived from a complementing 
 modality . thus , the information content in such a database is not 
 simply additive , but is likely to have an amplifying effect . large - scale database structures containing various brain disorders 
 can be then constructed with a common frame of reference , allowing 
 meta - analysis of data patterns distributed over several 
 modalities . obviously , such analyses require a well - developed 
 visual interface which allows the researchers to grasp the 
 relationship between data patterns in an intuitive way . we believe 
 that such a  visual database  has the potential to provide 
 physicians with additional important clues regarding the 
 pathological state of cortical areas and is likely to be of 
 relevance for the clinical management of epilepsy patients . as with every method , a few limitations need to be considered when 
 applying this method to images of children with intractable 
 epilepsy . as indicated above , we used young adults to create a 
 normal database , and this database was implicitly assumed to 
 characterize accurately normal tracer concentration patterns in 
 children . this approach is inevitable due to ethical guidelines 
 which sanction only the study of children who may derive direct 
 benefit from studies using administration of radioactive 
 substances . although the application of adult tracer concentration 
 patterns to children represents a potential source of error , it 
 appears that at least for fdg the tracer concentration patterns 
 might be very similar between children and adults . secondly , integration of brain pet and mr image volumes requires 
 their spatial alignment , a potential source of error . because the 
 brain is encased by the skull and can be therefore regarded as a 
 rigid body , alignment between the mr and pet brain image volumes 
 is likely to be accurate within a few millimeters , especially if 
 anatomical landmarks inside the brain are used . moreover , the 
 localization of the electrode grid on the cortical surface might 
 result in a maximal misregistration of about 3 mm . 
 thus , despite the fact that misalignment effects associated with 
 various alignment procedures might be additive , their total effect 
 is likely to be less than 5 mm . finally , the initial processing step of our method 
 requires the manual definition of cortical landmarks , guided by a 
 detailed surface rendering of cortical sulci and gyri . although 
 high - resolution mr images provide detailed features of the cortex , 
 reproducible definition of cortical landmarks is not trivial and 
 might contribute to the misalignment of presumably homotopic 
 cortical elements in different patients . in order to keep this 
 source of error to a minimum , we chose only anatomically 
 well - defined cortical features for our standard landmark set . nevertheless , the definition of cortical landmarks contributes the 
 largest error to possible misalignment between homotopic cortical 
 elements and future studies are warranted to assess their clinical 
 relevance . the creation of a multimodality extendable database 
 structure accessible through an advanced 3d visualization 
 interface holds promise of providing new insights into the 
 formation and identification of epileptic foci . a better 
 understanding of the relationship between functional and 
 electrophysiological data might lead to new approaches in epilepsy 
 surgery and will likely improve clinical management of a large 
 number of patients suffering from intractable epilepsy .
an important goal of software development in the medical field is the design of methods which are able to integrate information obtained from various imaging and nonimaging modalities into a cohesive framework in order to understand the results of qualitatively different measurements in a larger context . 
 moreover , it is essential to assess the various features of the data quantitatively so that relationships in anatomical and functional domains between complementing modalities can be expressed mathematically . 
 this paper presents a clinically feasible software environment for the quantitative assessment of the relationship among biochemical functions as assessed by pet imaging and electrophysiological parameters derived from intracranial eeg . 
 based on the developed software tools , quantitative results obtained from individual modalities can be merged into a data structure allowing a consistent framework for advanced data mining techniques and 3d visualization . 
 moreover , an effort was made to derive quantitative variables ( such as the spatial proximity index , spi ) characterizing the relationship between complementing modalities on a more generic level as a prerequisite for efficient data mining strategies . 
 we describe the implementation of this software environment in twelve children ( mean age 5.2  4.3 years ) with medically intractable partial epilepsy who underwent both high - resolution structural mr and functional pet imaging . 
 our experiments demonstrate that our approach will lead to a better understanding of the mechanisms of epileptogenesis and might ultimately have an impact on treatment 
 . moreover , our software environment holds promise to be useful in many other neurological disorders , where integration of multimodality data is crucial for a better understanding of the underlying disease mechanisms .